Pith. sign in

REVIEW 2 major objections 2 minor 192 references

Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability

T0 review · 2 major / 2 minor · reviewed 2026-05-11 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A unified architecture-lifecycle framework secures computer-use agents by grounding reliability in deployment realities.

desk verdict The paper gives a clean organizational framework for CUA security by splitting architecture into Perception-Decision-Execution and lifecycle into Creation-Deployment-Operation-Maintenance, but offers only re-categorization of prior work with no new tests or proofs. read the letter →

arxiv 2605.07110 v1 submitted 2026-05-08 cs.CL cs.SE

Zejian Chen , Zhanyuan Liu , Chaozhuo Li , Mengxiang Han , Songyang Liu , Litian Zhang , Feng Gao , Yiming Hei
show 1 more author
Xi Zhang
This is my paper · ORCID
classification cs.CLcs.SE
keywords computer-useagentsreliabilityframeworkagentsecurityperceptiondecisionexecutionlifecyclestagesdeploymentgroundingcontroloversightfailureanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that reliability for computer-use agents in real environments like browsers and desktops depends on more than task success, involving perception errors, planning drift, and permission scopes. It establishes a framework that combines an architectural view of Perception, Decision, and Execution layers transforming observations into actions with a lifecycle view of Creation, Deployment, Operation, and Maintenance stages for learning priors and preserving assurance. This synthesis of systems and studies distinguishes visible failures from their enabling conditions and maps surfaces for control oversight. A reader would care because without such grounding, agents risk misalignment with user intent in actual deployments where authority-bearing actions occur.

What carries the argument

The architecture-lifecycle framework that couples three layers (Perception, Decision, Execution) transforming observations into actions with four stages (Creation, Deployment, Operation, Maintenance) where conditions are set and assurance maintained.

What would settle it

Applying the framework to a collection of current computer-use agent systems and finding that it fails to reveal any additional intervention surfaces or control points beyond what prior surveys already noted would show the framework adds no new distinguishing power.

Watch

Extended reading notes

Core claim

The article develops an architecture-lifecycle framework for deployment-grounded reliability in computer-use agents. The architectural view analyzes Perception, Decision, and Execution as coupled layers that transform software observations into authority-bearing actions. The lifecycle view examines Creation, Deployment, Operation, and Maintenance as stages in which priors are learned, tools and permissions are bound, runtime trajectories are stressed, and assurance must be preserved under drift. Using this lens, the analysis synthesizes representative systems, benchmarks, and security studies to distinguish where failures become visible from where enabling conditions are introduced and to 1.

Load-bearing premise

That reviewing and synthesizing existing systems, benchmarks, and studies through these specific layers and stages will clearly separate visible failures from their enabling conditions without needing original experiments to validate the framework.

Editorial extensions

If this is right

  • Control oversight can target specific intervention surfaces identified across layers and stages.
  • Assurance preservation under drift becomes possible through maintenance stage analysis.
  • Open challenges like controllable grounding and safe authority binding are highlighted for future work.
  • Privacy-preserving memory and mixed-trust runtime defense emerge as key needs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Developers could use the framework to audit new computer-use agents for hidden permission risks before deployment.
  • The approach might extend to creating benchmarks that evaluate agents across full lifecycles rather than isolated tasks.
  • Similar layered views could apply to other autonomous systems beyond software agents, such as robotic controllers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims to develop a unified architecture-lifecycle framework for deployment-grounded reliability in computer-use agents (CUAs). The architectural view decomposes agents into coupled Perception, Decision, and Execution layers that convert observations into authority-bearing actions. The lifecycle view organizes the process into Creation, Deployment, Operation, and Maintenance stages where priors are acquired, tools/permissions bound, trajectories stressed, and assurance maintained under drift. Applying this lens to existing systems, benchmarks, and security studies, the work distinguishes locations where failures become visible from where enabling conditions are introduced and identifies recurring intervention surfaces for oversight. OpenClaw serves only as a public motivating example of an open deployment pattern.

Significance. If the framework provides a coherent organizational lens that reveals non-obvious mappings between architectural layers, lifecycle stages, and control surfaces, it could aid researchers and practitioners in systematically addressing reliability and security gaps in real-world CUA deployments beyond task-success metrics. The explicit connection of capability formation to authority exposure and failure manifestation is a potentially useful synthesis, though its value hinges on whether the distinctions prove actionable in subsequent empirical work.

major comments (2)
  1. [Abstract / synthesis of representative systems] Abstract and synthesis description: The central claim that the architecture-lifecycle lens 'distinguishes where failures become visible from where their enabling conditions are introduced' and thereby 'maps recurring intervention surfaces' rests entirely on qualitative re-categorization of prior literature. No formal categorization protocol, inter-annotator agreement, or controlled comparison against existing survey taxonomies is provided to establish that the distinctions are novel or practically enabling.
  2. [Lifecycle view] Lifecycle view description: The mapping of stages (Creation through Maintenance) to specific reliability mechanisms (prior learning, permission binding, drift handling) is asserted conceptually, but the manuscript supplies no concrete, traceable examples from the cited benchmarks or security studies demonstrating how this mapping uncovers intervention surfaces that prior analyses missed.
minor comments (2)
  1. [Abstract] The abstract contains multiple typographical and formatting issues, including 'flesystems' (should be 'filesystems'), missing spaces after commas, 'Creation.Deployment' (period instead of comma), and hyphenated 'author-ity'.
  2. [Introduction / motivating example] The precise scope and depth of the OpenClaw example should be stated explicitly in the main text to avoid any implication that it constitutes an internal case study.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below, providing clarifications on the scope of our qualitative framework and committing to revisions that strengthen the presentation of mappings and examples.

read point-by-point responses

  1. Referee: [Abstract / synthesis of representative systems] Abstract and synthesis description: The central claim that the architecture-lifecycle lens 'distinguishes where failures become visible from where their enabling conditions are introduced' and thereby 'maps recurring intervention surfaces' rests entirely on qualitative re-categorization of prior literature. No formal categorization protocol, inter-annotator agreement, or controlled comparison against existing survey taxonomies is provided to establish that the distinctions are novel or practically enabling.

    Authors: We agree that the synthesis is qualitative and does not employ a formal categorization protocol, inter-annotator agreement, or controlled comparison to existing taxonomies. The framework is offered as a conceptual organizational lens to connect architectural layers, lifecycle stages, and control surfaces, rather than as an empirically derived taxonomy. The distinctions emerge from re-examining cited systems, benchmarks, and security studies through this coupled structure, which prior surveys have not organized in this way. We will revise the abstract, introduction, and synthesis section to explicitly characterize the approach as qualitative and include a summary table of layer-stage-intervention mappings to improve transparency and actionability. revision: partial

  2. Referee: [Lifecycle view] Lifecycle view description: The mapping of stages (Creation through Maintenance) to specific reliability mechanisms (prior learning, permission binding, drift handling) is asserted conceptually, but the manuscript supplies no concrete, traceable examples from the cited benchmarks or security studies demonstrating how this mapping uncovers intervention surfaces that prior analyses missed.

    Authors: The referee is correct that while the manuscript applies the lens to representative works, more explicit tracing of individual examples would better demonstrate novel distinctions. The synthesis section currently references studies on perception errors, tool mediation, and runtime oversight, but does not always isolate how a specific enabling condition in one stage manifests in another. In revision we will add a new subsection with 2-3 traceable examples (e.g., linking a cited agent security study on permission overreach in Deployment to observable failures in Operation) to show intervention surfaces that were not foregrounded in the original analyses. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: conceptual framework is a non-derivational synthesis

full rationale

The manuscript proposes an architecture-lifecycle framework consisting of Perception-Decision-Execution layers and Creation-Deployment-Operation-Maintenance stages. It applies this lens to synthesize representative systems, benchmarks, and studies from prior literature, distinguishing failure visibility from enabling conditions via author-led categorization. No equations, derivations, fitted parameters, predictions, uniqueness theorems, or self-citations appear in the text. OpenClaw is explicitly limited to a public motivating example. The central claim is an interpretive organizational lens rather than a reduction of any result to its inputs by construction, rendering the derivation chain self-contained with no load-bearing circular steps.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The framework rests on domain assumptions about how CUAs operate in real environments and how failures manifest across layers and stages; no free parameters or new invented entities are introduced.

assumptions (2)
  • domain assumption Perception, Decision, and Execution function as coupled layers that transform software observations into authority-bearing actions.
    Invoked in the architectural view section of the abstract.
  • domain assumption Reliability and security issues arise and can be addressed across Creation, Deployment, Operation, and Maintenance stages.
    Invoked in the lifecycle view section of the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability." pith.science (2026). https://pith.science/paper/2605.07110

@misc{pith2026260507110,
  author       = {Pith},
  title        = {Pith review of: Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2605.07110}},
  note         = {Machine review of arXiv:2605.07110}
}
read the original abstract

Computer-use agents(CUAs)are moving frombounded benchmarks toward real software environments, wherethey operate browsers, desktops, mobile applications, flesystems,terminals, and tool backends. In such settings, reliability isno longer captured by task success alone: perception errors,planning drift, memory use, tool mediation, permission scope,and runtime oversight jointly determine whether agent actionsremain aligned with user intent, Existing surveys organize theCUA landscape by methods, platforms, benchmarks, or securitythreats, but less explicitly connect capability formation, author-ity exposure, failure manifestation, and control placement. Toaddress this gap, the article develops an architecture-lifecycleframework for deployment-grounded reliability in CUAs. Thearchitectural view analyzes Perception, Decision, and Executionas coupled layers that transform software observations intoauthority-bearing actions, The lifecycle view examines Creation.Deployment, Operation, and Maintenance as stages in which priorsare learned, tools and permissions are bound, runtime trajecto.ries are stressed, and assurance must be preserved under drift.Using this lens, the analysis synthesizes representative systems,benchmarks, and security/privacy studies; distinguishes wherefailures become visible from where their enabling conditions areintroduced, and maps recurring intervention surfaces for controloversight, and assurance. OpenClaw is used only as a public moti.vating example of an open deployment pattern, not as a verifedinternal case study. The conclusion highlights open challengesin controllable grounding, long-horizon constraint preservation,safe authority binding, mixed-trust runtime defense, privacy-preserving memory,and continual assurance.

Figures

Figures reproduced from arXiv: 2605.07110 by the authors.

Figure 1
Figure 1. Tri-layer pressure map for deployed CUAs. The figure frames deployed CUA behavior as a recurrent control loop between user goals and constraints [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Revised four-stage lifecycle framework for deployed CUAs. Each stage is organized into three aligned bands: the [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Lifecycle-aligned CUA threats and controls. The figure serves as an intervention map: for each stage, it identifies the salient threat surfaces, the system [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

192 extracted references · 192 canonical work pages

  1. [1]

    OpenCUA: Open foundations for computer-use agents.arXiv preprint arXiv:2508.09123,

    Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, et al. Opencua: Open foundations for computer-use agents.arXiv preprint arXiv:2508.09123, 2025

    work page arXiv 2025

  2. [2]

    arXiv preprint arXiv:2510.02250 , year=

    Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, and Xin Eric Wang. The unreasonable effectiveness of scaling agents for computer use.arXiv preprint arXiv:2510.02250, 2025

    work page arXiv 2025

  3. [3]

    arXiv preprint arXiv:2602.16855 , year=

    Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, et al. Mobile-agent-v3. 5: Multi-platform fundamental gui agents.arXiv preprint arXiv:2602.16855, 2026

    work page arXiv 2026

  4. [4]

    arXiv preprint arXiv:2512.22047 , year=

    Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, et al. Mai-ui technical report: Real-world centric foundation gui agents. arXiv preprint arXiv:2512.22047, 2025

    work page arXiv 2025

  5. [5]

    arXiv preprint arXiv:2512.15431 , year=

    Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, et al. Step-gui technical report.arXiv preprint arXiv:2512.15431, 2025

    work page arXiv 2025

  6. [6]

    arXiv2510.19949(2025).https : / / doi

    Mathieu Andreux, M ¨art Bakler, Yanael Barbier, Hamza Benchekroun, Emilien Bir ´e, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Aleix Cambray, et al. Surfer 2: The next generation of cross- platform computer use agents.arXiv preprint arXiv:2510.19949, 2025

    work page arXiv 2025

  7. [7]

    MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

    Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, et al. Molmoweb: Open visual web agent and open data for the open web.arXiv preprint arXiv:2604.08516, 2026

    work page Pith review arXiv 2026

  8. [8]

    Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    work page 2023

Show all 192 references
  1. [9]

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  2. [10]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. InProceedings of the 62nd Annual Meeting of the Associatio...

    2024

  3. [11]

    Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, L´eo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    2024

  4. [12]

    Workarena++: Towards composi- tional planning and reasoning-based common knowledge work tasks

    L ´eo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault L De Chezelles, Quentin Cappart, Nicolas Chapados, Alexan- dre Lacoste, and Alexandre Drouin. Workarena++: Towards composi- tional planning and reasoning-based common knowledge work tasks. Advances in Neural ...

    2024

  5. [13]

    Webforge: Breaking the realism-reproducibility-scalability trilemma in browser agent benchmark.arXiv preprint arXiv:2604.10988, 2026

    Peng Yuan, Yuyang Yin, Yuxuan Cai, and Zheng Wei. Webforge: Breaking the realism-reproducibility-scalability trilemma in browser agent benchmark.arXiv preprint arXiv:2604.10988, 2026

    2026 arXiv

  6. [14]

    Clawbench: Can ai agents complete everyday online tasks? arXiv preprint arXiv:2604.08523, 2026

    Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, et al. Clawbench: Can ai agents complete everyday online tasks? arXiv preprint arXiv:2604.08523, 2026

    2026 arXiv

  7. [15]

    Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation.arXiv preprint arXiv:2604.08455, 2026

    Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, et al. Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation.arXiv preprint arXiv:2604.08455, 2026

    2026 arXiv

  8. [16]

    Pspa-bench: A personalized benchmark for smartphone gui agent.arXiv preprint arXiv:2603.29318, 2026

    Hongyi Nie, Xunyuan Liu, Yudong Bai, Yaqing Wang, Yang Liu, Quan- ming Yao, and Zhen Wang. Pspa-bench: A personalized benchmark for smartphone gui agent.arXiv preprint arXiv:2603.29318, 2026

    2026

  9. [17]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Sy...

    2024

  10. [18]

    Colorbrowseragent: Complex long-horizon browser agent with adaptive knowledge evolution

    Jihong Wang, Jiamu Zhou, Weiming Zhang, Teng Wang, Weiwen Liu, Zhuosheng Zhang, Xingyu Lou, Weinan Zhang, HUARONG DENG, and Jun Wang. Colorbrowseragent: Complex long-horizon browser agent with adaptive knowledge evolution. InThe 64th Annual Meeting of the Association for Compu...

    2026

  11. [19]

    GraphPilot: GUI Task Automa- tion with One-Step LLM Reasoning Powered by Knowledge Graph, jan 2026

    Mingxian Yu, Siqi Luo, and Xu Chen. GraphPilot: GUI Task Automa- tion with One-Step LLM Reasoning Powered by Knowledge Graph, jan 2026. Journal of Intelligent Computing and Networking

    2026

  12. [20]

    Showui-aloha: Human-taught gui agent.arXiv preprint arXiv:2601.07181, 2026

    Yichun Zhang, Xiangwu Guo, Yauhong Goh, Jessica Hu, Zhiheng Chen, Xin Wang, Difei Gao, and Mike Zheng Shou. Showui-aloha: Human-taught gui agent.arXiv preprint arXiv:2601.07181, 2026

    2026

  13. [21]

    Gui agents: A survey

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey. InFindings of the Association for Computational Linguistics: ACL 2025, pages 22522–22538, 2025

    2025

  14. [22]

    Os agents: A survey on mllm-based agents for computer, phone and browser use

    Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for computer, phone and browser use. InProceedings of the 63rd Annual Meeting of the Association for Computati...

    2025

  15. [23]

    A survey on (m) llm-based gui agents.arXiv preprint arXiv:2504.13865, 2025

    Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, et al. A survey on (m) llm-based gui agents.arXiv preprint arXiv:2504.13865, 2025

    2025

  16. [24]

    Gui agents with foundation models: A comprehensive survey.arXiv preprint arXiv:2411.04890, 2024

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. Gui agents with foundation models: A comprehensive survey.arXiv preprint arXiv:2411.04890, 2024

    2024

  17. [25]

    Generalist virtual agents: A survey on autonomous agents across digital platforms.arXiv preprint arXiv:2411.10943, 2024

    Minghe Gao, Wendong Bu, Bingchen Miao, Yang Wu, Yunfei Li, Juncheng Li, Siliang Tang, Qi Wu, Yueting Zhuang, and Meng Wang. Generalist virtual agents: A survey on autonomous agents across digital platforms.arXiv preprint arXiv:2411.10943, 2024

    2024

  18. [26]

    Towards trustworthy gui agents: A survey.arXiv preprint arXiv:2503.23434, 2025

    Yucheng Shi, Wenhao Yu, Wenlin Yao, Wenhu Chen, and Ninghao Liu. Towards trustworthy gui agents: A survey.arXiv preprint arXiv:2503.23434, 2025

    2025

  19. [27]

    A survey on gui agents with foun- dation models enhanced by reinforcement learning.arXiv preprint arXiv:2504.20464, 2025

    Jiahao Li and Kaer Huang. A survey on gui agents with foun- dation models enhanced by reinforcement learning.arXiv preprint arXiv:2504.20464, 2025

    2025

  20. [28]

    Llm-powered gui agents in phone automation: Surveying progress and prospects.arXiv preprint arXiv:2504.19838, 2025

    Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, et al. Llm-powered gui agents in phone automation: Surveying progress and prospects.arXiv preprint arXiv:2504.19838, 2025

    2025

  21. [29]

    A survey of webagents: Towards next-generation ai agents for web automation with large foundation models

    Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, et al. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In31st ACM SIGKDD Conference on Knowl...

    2025

  22. [30]

    A survey on the safety and security threats of computer-using agents: Jarvis or ultron?arXiv preprint arXiv:2505.10924, 2025

    Ada Chen, Yongjiang Wu, Junyuan Zhang, Jingyu Xiao, Shu Yang, Jen- tse Huang, Kun Wang, Wenxuan Wang, and Shuai Wang. A survey on the safety and security threats of computer-using agents: Jarvis or ultron?arXiv preprint arXiv:2505.10924, 2025

    2025 arXiv

  23. [31]

    Openclaw — personal ai assistant

    OpenClaw. Openclaw — personal ai assistant. https://openclaw.ai/,

  24. [32]

    Website, accessed: 2026-04-11

    2026

  25. [33]

    Openclaw, moltbook and the future of ai agents

    Aili McConnon. Openclaw, moltbook and the future of ai agents. https://www.ibm.com/think/news/clawdbot-ai-agent-testing-limits-ver tical-integration, 2026. IBM Think article, accessed: 2026-04-11

    2026

  26. [34]

    Clawgui: A unified framework for training, evaluating, and deploying gui agents.arXiv preprint arXiv:2604.11784, 2026

    Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Clawgui: A unified framework for training, evaluating, and deploying gui agents.arXiv preprint arXiv:2604.11784, 2026

    2026 arXiv

  27. [35]

    LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent

    Bin Kang, Shaoguo Wen, Yifei Bi, Shunlong Wu, Xinbin Yuan, Rui Shao, Junle Wang, and Zhuotao Tian. LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent. InInternational Conference on Learning Representations, 2026. Poster

    2026

  28. [36]

    Webarena-infinity: Generating browser environments with verifiable tasks at scale.shuyanzhou.com, March 2026

    Shuyan Zhou. Webarena-infinity: Generating browser environments with verifiable tasks at scale.shuyanzhou.com, March 2026

    2026

  29. [37]

    Webpii: Benchmarking visual pii detection for computer- use agents.arXiv preprint arXiv:2603.17357, 2026

    Nathan Zhao. Webpii: Benchmarking visual pii detection for computer- use agents.arXiv preprint arXiv:2603.17357, 2026

    2026

  30. [38]

    Guide: Interpretable gui agent evaluation via hierarchical diagnosis.arXiv preprint arXiv:2604.04399, 2026

    Yuwen Zhai, Runze Li, Liang Wang, Nian Shi, Liwu Xu, Wei Zhang, Ran Lin, Bo Xu, and Benlei Cui. Guide: Interpretable gui agent evaluation via hierarchical diagnosis.arXiv preprint arXiv:2604.04399, 2026

    2026 arXiv

  31. [39]

    Cuarewardbench: A benchmark for evaluating reward models on computer-using agent

    Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, et al. Cuarewardbench: A benchmark for evaluating reward models on computer-using agent. arXiv preprint arXiv:2510.18596, 2025

    2025

  32. [40]

    Gym-anything: Turn any software into an agent environment.arXiv preprint arXiv:2604.06126, 2026

    Pranjal Aggarwal, Graham Neubig, and Sean Welleck. Gym-anything: Turn any software into an agent environment.arXiv preprint arXiv:2604.06126, 2026

    2026 arXiv

  33. [41]

    Os-harm: A benchmark for measuring safety of computer use agents.arXiv preprint arXiv:2506.14866, 2025

    Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. Os-harm: A benchmark for measuring safety of computer use agents.arXiv preprint arXiv:2506.14866, 2025

    2025

  34. [42]

    Wainjectbench: Benchmarking prompt injection detections for web agents.arXiv preprint arXiv:2510.01354, 2025

    Yinuo Liu, Ruohan Xu, Xilong Wang, Yuqi Jia, and Neil Zhenqiang Gong. Wainjectbench: Benchmarking prompt injection detections for web agents.arXiv preprint arXiv:2510.01354, 2025

    2025

  35. [43]

    Risky-bench: Probing agentic safety risks under real-world deployment.arXiv preprint arXiv:2602.03100, 2026

    Jingnan Zheng, Yanzhen Luo, Jingjun Xu, Bingnan Liu, Yuxin Chen, Chenhang Cui, Gelei Deng, Chaochao Lu, Xiang Wang, An Zhang, et al. Risky-bench: Probing agentic safety risks under real-world deployment.arXiv preprint arXiv:2602.03100, 2026

    2026

  36. [44]

    The aegis protocol: A foundational security framework for autonomous ai agents

    Sai Teja Reddy Adapala and Yashwanth Reddy Alugubelly. The aegis protocol: A foundational security framework for autonomous ai agents. arXiv preprint arXiv:2508.19267, 2025

    2025

  37. [45]

    A survey of agentic ai and cybersecurity: Challenges, opportunities and use-case prototypes.arXiv preprint arXiv:2601.05293, 2026

    Sahaya Jestus Lazer, Kshitiz Aryal, Maanak Gupta, and Elisa Bertino. A survey of agentic ai and cybersecurity: Challenges, opportunities and use-case prototypes.arXiv preprint arXiv:2601.05293, 2026

    2026

  38. [46]

    Os-copilot: Towards generalist computer agents with self-improvement.arXiv preprint arXiv:2402.07456, 2024

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumi- anze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement.arXiv preprint arXiv:2402.07456, 2024

    2024

  39. [47]

    Coact-1: Computer-using agents with coding as actions.arXiv preprint arXiv:2508.03923, 2025

    Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, et al. Coact-1: Computer-using agents with coding as actions.arXiv preprint arXiv:2508.03923, 2025

    2025

  40. [48]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hong- ming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume ...

    2024

  41. [49]

    Camels can use computers too: System-level security for computer use agents.arXiv preprint arXiv:2601.09923, 2026

    Hanna Foerster, Robert Mullins, Tom Blanchard, Nicolas Papernot, Kristina Nikoli ´c, Florian Tram `er, Ilia Shumailov, Cheng Zhang, and Yiren Zhao. Camels can use computers too: System-level security for computer use agents.arXiv preprint arXiv:2601.09923, 2026

    2026

  42. [50]

    Riosworld: Benchmarking the risk of multimodal computer-use agents.arXiv preprint arXiv:2506.00618, 2025

    Jingyi Yang, Shuai Shao, Dongrui Liu, and Jing Shao. Riosworld: Benchmarking the risk of multimodal computer-use agents.arXiv preprint arXiv:2506.00618, 2025

    2025

  43. [51]

    Ferret-ui: Grounded mobile ui understanding with multimodal llms

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. InEuropean Conference on Computer Vision, pages 240–255. Springer, 2024

    2024

  44. [52]

    Trishul: Towards region identification and screen hierarchy understanding for large vlm based gui agents

    Kunal Singh, Shreyas Singh, and Mukund Khanna. Trishul: Towards region identification and screen hierarchy understanding for large vlm based gui agents. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 170–179, 2025

    2025

  45. [53]

    Gui-bee: Align gui action grounding to novel environments via autonomous exploration

    Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-bee: Align gui action grounding to novel environments via autonomous exploration. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33249– 33266, 2025

    2025

  46. [54]

    Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning.arXiv preprint arXiv:2505.12370, 2025

    Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, and Bo Li. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning.arXiv preprint arXiv:2505.12370, 2025

    2025

  47. [55]

    Vimo: A generative visual gui world model for app agent.arXiv preprint arXiv:2504.13936, 2025

    Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye Hao, Jun Wang, and Kun Shao. Vimo: A generative visual gui world model for app agent.arXiv preprint arXiv:2504.13936, 2025

    2025

  48. [56]

    Secagent: Efficient mobile gui agent with semantic context.arXiv preprint arXiv:2603.08533, 2026

    Yiping Xie, Song Chen, Jingxuan Xing, Wei Jiang, Zekun Zhu, Yingyao Wang, Pi Bu, Jun Song, Yuning Jiang, and Bo Zheng. Secagent: Efficient mobile gui agent with semantic context.arXiv preprint arXiv:2603.08533, 2026

    2026

  49. [57]

    Webatlas: An llm agent with experience-driven memory and action simulation.arXiv preprint arXiv:2510.22732, 2025

    Jiali Cheng, Anjishnu Kumar, Roshan Lal, Rishi Rajasekaran, Hani Ramezani, Omar Zia Khan, Oleg Rokhlenko, Sunny Chiu-Webster, Gang Hua, and Hadi Amiri. Webatlas: An llm agent with experience-driven memory and action simulation.arXiv preprint arXiv:2510.22732, 2025

    2025

  50. [58]

    Litewebagent: The open-source suite for vlm-based web-agent applications

    Danqing Zhang, Balaji Rama, Jingyi Ni, Shiying He, Fu Zhao, Kunyu Chen, Arnold Chen, and Junyu Cao. Litewebagent: The open-source suite for vlm-based web-agent applications. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Comp...

    2025

  51. [59]

    A real-world webagent with planning, long context understanding, and program synthesis

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023

    2023

  52. [60]

    When openclaw ai agents teach each other: Peer learning patterns in the moltbook community.arXiv preprint arXiv:2602.14477, 2026

    Eason Chen, Ce Guan, Ahmed Elshafiey, Zhonghao Zhao, Joshua Zekeri, Afeez Edeifo Shaibu, and Emmanuel Osadebe Prince. When openclaw ai agents teach each other: Peer learning patterns in the moltbook community.arXiv preprint arXiv:2602.14477, 2026

    2026

  53. [61]

    Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadal- lah. Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    2024

  54. [62]

    Winclick: Gui grounding with multimodal large language models.arXiv preprint arXiv:2503.04730, 2025

    Zheng Hui, Yinheng Li, Tianyi Chen, Colby Banbury, Kazuhito Koishida, et al. Winclick: Gui grounding with multimodal large language models.arXiv preprint arXiv:2503.04730, 2025

    2025

  55. [63]

    Adazoom-gui: Adaptive zoom-based gui grounding with instruction refinement.arXiv preprint arXiv:2603.17441, 2026

    Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, et al. Adazoom-gui: Adaptive zoom-based gui grounding with instruction refinement.arXiv preprint arXiv:2603.17441, 2026

    2026

  56. [64]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. arXiv preprint arXiv:2306.07863, 2023

    2023

  57. [65]

    Ufo: A ui-focused agent for windows os interaction

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Comp...

    2025

  58. [66]

    Ee-mcp: Self-evolving mcp-gui agents via automated environment generation and experience learning.arXiv preprint arXiv:2604.09815, 2026

    Tiantian He, Yihang Chen, Keyue Jiang, Ka Yiu Lee, Kaiwen Zhou, Kun Shao, and Shuai Wang. Ee-mcp: Self-evolving mcp-gui agents via automated environment generation and experience learning.arXiv preprint arXiv:2604.09815, 2026

    2026 arXiv

  59. [67]

    Treecua: Effi- ciently scaling gui automation with tree-structured verifiable evolution

    Deyang Jiang, Jing Huang, Xuanle Zhao, Lei Chen, Liming Zheng, Fanfan Liu, Haibo Qiu, Peng Shi, and Zhixiong Zeng. Treecua: Effi- ciently scaling gui automation with tree-structured verifiable evolution. arXiv preprint arXiv:2602.09662, 2026

    2026

  60. [68]

    Appagentx: Evolving gui agents as proficient smartphone users.arXiv preprint arXiv:2503.02268, 2025

    Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, Joey Tianyi Zhou, and Chi Zhang. Appagentx: Evolving gui agents as proficient smartphone users.arXiv preprint arXiv:2503.02268, 2025

    2025

  61. [69]

    Gui-360◦: A comprehensive dataset and benchmark for computer-using agents.arXiv preprint arXiv:2511.04307, 2025

    Jian Mu, Chaoyun Zhang, Chiming Ni, Lu Wang, Bo Qiao, Kartik Mathur, Qianhui Wu, Yuhang Xie, Xiaojun Ma, Mengyu Zhou, et al. Gui-360◦: A comprehensive dataset and benchmark for computer-using agents.arXiv preprint arXiv:2511.04307, 2025

    2025

  62. [70]

    Showui-π: Flow-based generative models as gui dexterous hands.arXiv preprint arXiv:2512.24965, 2025

    Siyuan Hu, Kevin Qinghong Lin, and Mike Zheng Shou. Showui-π: Flow-based generative models as gui dexterous hands.arXiv preprint arXiv:2512.24965, 2025

    2025

  63. [71]

    Polyskill: Learning generalizable skills through polymorphic abstraction.arXiv preprint arXiv:2510.15863, 2025

    Simon Yu, Gang Li, Weiyan Shi, and Peng Qi. Polyskill: Learning generalizable skills through polymorphic abstraction.arXiv preprint arXiv:2510.15863, 2025

    2025

  64. [72]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2024

  65. [73]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception.arXiv preprint arXiv:2401.16158, 2024

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception.arXiv preprint arXiv:2401.16158, 2024

    2024

  66. [74]

    Screenai: A vision-language model for ui and infographics understanding.arXiv preprint arXiv:2402.04615, 2024

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor C˘arbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding.arXiv preprint arXiv:2402.04615, 2024

    2024

  67. [75]

    Moving beyond sparse grounding with complete screen parsing supervision.arXiv preprint arXiv:2602.14276, 2026

    A Said Gurbuz, Sunghwan Hong, Ahmed Nassar, Marc Pollefeys, and Peter Staar. Moving beyond sparse grounding with complete screen parsing supervision.arXiv preprint arXiv:2602.14276, 2026

    2026 arXiv

  68. [76]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14281–14290, 2024

    2024

  69. [77]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Au- gustus Odena, Arushi Somani, and Sa ˘gnak Tas ¸ırlar. Introducing our multimodal models, 2023

    2023

  70. [78]

    A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 6:3, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and J Qwen-VL Zhou. A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 6:3, 2023

    2023 arXiv

  71. [79]

    Afragent: An adaptive feature renormalization based high resolution aware gui agent

    Neeraj Anand, Rishabh Jain, Sohan Patnaik, Balaji Krishnamurthy, and Mausoom Sarkar. Afragent: An adaptive feature renormalization based high resolution aware gui agent. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1147– 1158, 2026

    2026

  72. [80]

    ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786. ACM, 2025

    2025

  73. [81]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Confer- ence on Human Factors in Computing Systems, pages 1–20, 2025

    2025

  74. [82]

    Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection.arXiv preprint arXiv:2501.04575, 2025

    Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection.arXiv preprint arXiv:2501.04575, 2025

    2025

  75. [83]

    Androtmem: From interaction trajectories to anchored memory in long- horizon gui agents.arXiv preprint arXiv:2603.18429, 2026

    Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, et al. Androtmem: From interaction trajectories to anchored memory in long- horizon gui agents.arXiv preprint arXiv:2603.18429, 2026

    2026

  76. [84]

    Gui-reflection: Empowering multimodal gui models with self-reflection behavior.arXiv preprint arXiv:2506.08012, 2025

    Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, and Ziwei Liu. Gui-reflection: Empowering multimodal gui models with self-reflection behavior.arXiv preprint arXiv:2506.08012, 2025

    2025

  77. [85]

    Mobileuse: A gui agent with hierarchical reflection for autonomous mobile operation.arXiv preprint arXiv:2507.16853, 2025

    Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kouni- anhua Du, Xingyu Lou, Qiuying Peng, Jun Wang, and Weinan Zhang. Mobileuse: A gui agent with hierarchical reflection for autonomous mobile operation.arXiv preprint arXiv:2507.16853, 2025

    2025

  78. [86]

    Lps-bench: Benchmarking safety awareness of computer-use agents in long-horizon planning under benign and adversarial scenarios

    Tianyu Chen, Chujia Hu, Ge Gao, Dongrui Liu, Xia Hu, and Wenjie Wang. Lps-bench: Benchmarking safety awareness of computer-use agents in long-horizon planning under benign and adversarial scenarios. arXiv preprint arXiv:2602.03255, 2026

    2026

  79. [87]

    When users change their mind: Evaluating interruptible agents in long-horizon web navigation.arXiv preprint arXiv:2604.00892, 2026

    Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang, Yankai Chen, Yue Zhou, Hanrong Zhang, Yaozu Wu, Liancheng Fang, Zhengyao Gu, Zhen Zhang, et al. When users change their mind: Evaluating interruptible agents in long-horizon web navigation.arXiv preprint arXiv:2604.00892, 2026

    2026

  80. [88]

    Guide: A benchmark for under- standing and assisting users in open-ended gui tasks.arXiv preprint arXiv:2603.25864, 2026

    Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, and Juho Kim. Guide: A benchmark for under- standing and assisting users in open-ended gui tasks.arXiv preprint arXiv:2603.25864, 2026

    2026

  81. [89]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    2022

  82. [90]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    2023

  83. [91]

    Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634– 8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634– 8652, 2023

    2023

  84. [92]

    You only look at screens: Multimodal chain-of-action agents

    Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. InFindings of the Association for Computational Linguistics: ACL 2024, pages 3132–3149, 2024

    2024

  85. [93]

    Reinagent: A context-aware gui agent enabling human-in-the- loop mobile task navigation.arXiv preprint arXiv:2510.07988, 2025

    Haitao Jia, Ming He, Zimo Yin, Likang Wu, Jianping Fan, and Jitao Sang. Reinagent: A context-aware gui agent enabling human-in-the- loop mobile task navigation.arXiv preprint arXiv:2510.07988, 2025

    2025

  86. [94]

    Backtrackagent: Enhancing gui agent with error detection and backtracking mechanism

    Qinzhuo Wu, Pengzhi Gao, Wei Liu, and Jian Luan. Backtrackagent: Enhancing gui agent with error detection and backtracking mechanism. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4250–4272, 2025

    2025

  87. [95]

    Building a stable planner: An extended finite state machine based planning module for mobile gui agent.arXiv preprint arXiv:2505.14141, 2025

    Fanglin Mo, Junzhe Chen, Haoxuan Zhu, and Xuming Hu. Building a stable planner: An extended finite state machine based planning module for mobile gui agent.arXiv preprint arXiv:2505.14141, 2025

    2025

  88. [96]

    Don’t act blindly: Robust gui automation via action-effect verification and self-correction.arXiv preprint arXiv:2604.05477, 2026

    Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, et al. Don’t act blindly: Robust gui automation via action-effect verification and self-correction.arXiv preprint arXiv:2604.05477, 2026

    2026 arXiv

  89. [97]

    Agent-sama: State-aware mobile assistant

    Linqiang Guo, Wei Liu, Yi Wen Heng, Tse-Hsun Peter Chen, and Yang Wang. Agent-sama: State-aware mobile assistant. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29459– 29467, 2026

    2026

  90. [98]

    Cradle: Empowering foundation agents towards general computer control.arXiv preprint arXiv:2403.03186, 2024

    Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control.arXiv preprint arXiv:2403.03186, 2024

    2024

  91. [99]

    Androidenv: A reinforcement learning platform for android.arXiv preprint arXiv:2105.13231, 2021

    Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android.arXiv preprint arXiv:2105.13231, 2021

    2021

  92. [100]

    Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning.arXiv preprint arXiv:2509.02544, 2025

    Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning.arXiv preprint arXiv:2509.02544, 2025

    2025

  93. [101]

    Screenagent: A vi- sion language model-driven computer control agent.arXiv preprint arXiv:2402.07945, 2024

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vi- sion language model-driven computer control agent.arXiv preprint arXiv:2402.07945, 2024

    2024

  94. [102]

    Omegause: Building a general-purpose gui agent for autonomous task execution

    Le Zhang, Yixiong Xiao, Xinjiang Lu, Jingjia Cao, Yusai Zhao, Jingbo Zhou, Lang An, Zikan Feng, Wanxiang Sha, Yu Shi, et al. Omegause: Building a general-purpose gui agent for autonomous task execution. arXiv preprint arXiv:2601.20380, 2026

    2026

  95. [103]

    macosworld: A multilingual interactive benchmark for gui agents.arXiv preprint arXiv:2506.04135, 2025

    Pei Yang, Hai Ci, and Mike Zheng Shou. macosworld: A multilingual interactive benchmark for gui agents.arXiv preprint arXiv:2506.04135, 2025

    2025

  96. [104]

    Websp-eval: Evaluating web agents on website security and privacy tasks.arXiv preprint arXiv:2604.06367, 2026

    Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, and Kassem Fawaz. Websp-eval: Evaluating web agents on website security and privacy tasks.arXiv preprint arXiv:2604.06367, 2026

    2026 arXiv

  97. [105]

    Turing test on screen: A benchmark for mobile gui agent humanization.arXiv preprint arXiv:2604.09574, 2026

    Jiachen Zhu, Lingyu Yang, Rong Shan, Congmin Zheng, Zeyu Zheng, Weiwen Liu, Yong Yu, Weinan Zhang, and Jianghao Lin. Turing test on screen: A benchmark for mobile gui agent humanization.arXiv preprint arXiv:2604.09574, 2026

    2026 arXiv

  98. [106]

    Modeling distinct human interaction in web agents.arXiv preprint arXiv:2602.17588, 2026

    Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo, Venu Arvind Arangarajan, Tianyue Ou, Frank Xu, Shuyan Zhou, Graham Neubig, and Jeffrey P Bigham. Modeling distinct human interaction in web agents.arXiv preprint arXiv:2602.17588, 2026

    2026

  99. [107]

    Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    2023

  100. [108]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. InFindings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, 2024

    2024

  101. [109]

    Tongui: Internet-scale trajectories from multimodal web tutorials for general- ized gui agents.arXiv preprint arXiv:2504.12679, 2025

    Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li. Tongui: Internet-scale trajectories from multimodal web tutorials for general- ized gui agents.arXiv preprint arXiv:2504.12679, 2025

    2025

  102. [110]

    Webchain: A large-scale human- annotated dataset of real-world web interaction traces.arXiv preprint arXiv:2603.05295, 2026

    Sicheng Fan, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. Webchain: A large-scale human- annotated dataset of real-world web interaction traces.arXiv preprint arXiv:2603.05295, 2026

    2026 arXiv

  103. [111]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. InProceedings of the 63rd Annual Meeting of the Associ...

    2025

  104. [112]

    Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620, 2025

    Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620, 2025

    2025

  105. [113]

    Gui-r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458, 2025

    Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, and Xiaobo Xia. Gui-r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458, 2025

    2025

  106. [114]

    Progrm: Build better gui agents with progress rewards.arXiv preprint arXiv:2505.18121, 2025

    Danyang Zhang, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen, and Kai Yu. Progrm: Build better gui agents with progress rewards.arXiv preprint arXiv:2505.18121, 2025

    2025

  107. [115]

    Web-shepherd: Advancing prms for reinforcing web agents.arXiv preprint arXiv:2505.15277, 2025

    Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Se- ungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, et al. Web-shepherd: Advancing prms for reinforcing web agents.arXiv preprint arXiv:2505.15277, 2025

    2025

  108. [116]

    Ui-genie: A self-improving approach for iteratively boosting mllm-based mobile gui agents.arXiv preprint arXiv:2505.21496, 2025

    Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, et al. Ui-genie: A self-improving approach for iteratively boosting mllm-based mobile gui agents.arXiv preprint arXiv:2505.21496, 2025

    2025

  109. [117]

    Magicgui- rms: A multi-agent reward model system for self-evolving gui agents via automated feedback reflux.arXiv preprint arXiv:2601.13060, 2026

    Zecheng Li, Zhihui Cao, Wenke Huang, Yudong Zhang, Keying Qi, Rui Wang, Zeyu Zheng, Jian Zhao, Hao Zhu, Hengxin Wu, et al. Magicgui- rms: A multi-agent reward model system for self-evolving gui agents via automated feedback reflux.arXiv preprint arXiv:2601.13060, 2026

    2026

  110. [118]

    Video- based reward modeling for computer-use agents.arXiv preprint arXiv:2603.10178, 2026

    Linxin Song, Jieyu Zhang, Huanxin Sheng, Taiwei Shi, Gupta Rahul, Yang Liu, Ranjay Krishna, Jian Kang, and Jieyu Zhao. Video- based reward modeling for computer-use agents.arXiv preprint arXiv:2603.10178, 2026

    2026

  111. [119]

    Mcp security bench (msb): Benchmarking attacks against model context protocol in llm agents.arXiv preprint arXiv:2510.15994, 2025

    Dongsen Zhang, Zekun Li, Xu Luo, Xuannan Liu, Peipei Li, and Wenjun Xu. Mcp security bench (msb): Benchmarking attacks against model context protocol in llm agents.arXiv preprint arXiv:2510.15994, 2025

    2025

  112. [120]

    Les dissonances: Cross-tool harvesting and polluting in pool-of-tools empowered llm agents.arXiv preprint arXiv:2504.03111, 2025

    Zichuan Li, Jian Cui, Xiaojing Liao, and Luyi Xing. Les dissonances: Cross-tool harvesting and polluting in pool-of-tools empowered llm agents.arXiv preprint arXiv:2504.03111, 2025

    2025

  113. [121]

    Worldgui: An interactive benchmark for desktop gui automation from any starting point.arXiv preprint arXiv:2502.08047, 2025

    Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, and Mike Zheng Shou. Worldgui: An interactive benchmark for desktop gui automation from any starting point.arXiv preprint arXiv:2502.08047, 2025

    2025

  114. [122]

    Actio- nengine: From reactive to programmatic gui agents via state machine memory.arXiv preprint arXiv:2602.20502, 2026

    Hongbin Zhong, Fazle Faisal, Luis Franc ¸a, Tanakorn Leesataporn- wongsa, Adriana Szekeres, Kexin Rong, and Suman Nath. Actio- nengine: From reactive to programmatic gui agents via state machine memory.arXiv preprint arXiv:2602.20502, 2026

    2026

  115. [123]

    The amazing agent race: Strong tool users, weak navigators.arXiv preprint arXiv:2604.10261, 2026

    Zae Myung Kim, Dongseok Lee, Jaehyung Kim, Vipul Raheja, and Dongyeop Kang. The amazing agent race: Strong tool users, weak navigators.arXiv preprint arXiv:2604.10261, 2026

    2026 arXiv

  116. [124]

    Mobileworldbench: Towards semantic world modeling for mobile agents.arXiv preprint arXiv:2512.14014, 2025

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Mobileworldbench: Towards semantic world modeling for mobile agents.arXiv preprint arXiv:2512.14014, 2025

    2025

  117. [125]

    Webinject: Prompt injection attack to web agents

    Xilong Wang, John Bloch, Zedian Shao, Yuepeng Hu, Shuyan Zhou, and Neil Zhenqiang Gong. Webinject: Prompt injection attack to web agents. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2010–2030, 2025

    2025

  118. [126]

    In-context defense in computer agents: An empirical study.arXiv preprint arXiv:2503.09241, 2025

    Pei Yang, Hai Ci, and Mike Zheng Shou. In-context defense in computer agents: An empirical study.arXiv preprint arXiv:2503.09241, 2025

    2025

  119. [127]

    Evaluating the robustness of multimodal agents against active environmental injection attacks

    Yurun Chen, Xueyu Hu, Keting Yin, Juncheng Li, and Shengyu Zhang. Evaluating the robustness of multimodal agents against active environmental injection attacks. InProceedings of the 33rd ACM International Conference on Multimedia, pages 11648–11656, 2025

    2025

  120. [128]

    From assistant to double agent: Formalizing and benchmarking at- tacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412, 2026

    Yuhang Wang, Feiming Xu, Zheng Lin, Guangyu He, Yuzhe Huang, Haichang Gao, Zhenxing Niu, Shiguo Lian, and Zhaoxiang Liu. From assistant to double agent: Formalizing and benchmarking at- tacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412, 2026

    2026

  121. [129]

    Your agent, their asset: A real-world safety analysis of openclaw.arXiv preprint arXiv:2604.04759, 2026

    Zijun Wang, Haoqin Tu, Letian Zhang, Hardy Chen, Juncheng Wu, Xiangyan Liu, Zhenlong Yuan, Tianyu Pang, Michael Qizhe Shieh, Fengze Liu, et al. Your agent, their asset: A real-world safety analysis of openclaw.arXiv preprint arXiv:2604.04759, 2026

    2026 arXiv

  122. [130]

    Cora: Conformal risk- controlled agents for safeguarded mobile gui automation.arXiv preprint arXiv:2604.09155, 2026

    Yushi Feng, Junye Du, Qifan Wang, Zizhan Ma, Qian Niu, Yu- taka Matsuo, Long Feng, and Lequan Yu. Cora: Conformal risk- controlled agents for safeguarded mobile gui automation.arXiv preprint arXiv:2604.09155, 2026

    2026 arXiv

  123. [131]

    Magnet: Towards adaptive gui agents with memory-driven knowledge evolution

    Libo Sun, Jiwen Zhang, Siyuan Wang, and Zhongyu Wei. Magnet: Towards adaptive gui agents with memory-driven knowledge evolution. arXiv preprint arXiv:2601.19199, 2026

    2026

  124. [132]

    Ui-oceanus: Scaling gui agents with synthetic environmental dynamics.arXiv preprint arXiv:2604.02345, 2026

    Mengzhou Wu, Yuzhe Guo, Yuan Cao, Haochuan Lu, Songhe Zhu, Pingzhe Qu, Xin Chen, Kang Qin, Zhongpu Wang, Xiaode Zhang, et al. Ui-oceanus: Scaling gui agents with synthetic environmental dynamics.arXiv preprint arXiv:2604.02345, 2026

    2026 arXiv

  125. [133]

    Memgui-bench: Benchmarking memory of mobile gui agents in dynamic environments.arXiv preprint arXiv:2602.06075, 2026

    Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, et al. Memgui-bench: Benchmarking memory of mobile gui agents in dynamic environments.arXiv preprint arXiv:2602.06075, 2026

    2026

  126. [134]

    TimeWarp: Evaluating Web Agents by Revisiting the Past, mar 2026

    Md Farhan Ishmam and Kenneth Marino. TimeWarp: Evaluating Web Agents by Revisiting the Past, mar 2026. arXiv preprint arXiv:2603.04949

    2026

  127. [135]

    Vision2web: A hierarchical benchmark for visual website development with agent verification.arXiv preprint arXiv:2603.26648, 2026

    Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, and Jie Tang. Vision2web: A hierarchical benchmark for visual website development with agent verification.arXiv preprint arXiv:2603.26648, 2026

    2026

  128. [136]

    Webtest- bench: Evaluating computer-use agents towards end-to-end automated web testing.arXiv preprint arXiv:2603.25226, 2026

    Fanheng Kong, Jingyuan Zhang, Yang Yue, Chenxi Sun, Yang Tian, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Jun Du, et al. Webtest- bench: Evaluating computer-use agents towards end-to-end automated web testing.arXiv preprint arXiv:2603.25226, 2026

    2026

  129. [137]

    Guitester: Enabling gui agents for exploratory defect discovery.arXiv preprint arXiv:2601.04500, 2026

    Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, and Jitao Sang. Guitester: Enabling gui agents for exploratory defect discovery.arXiv preprint arXiv:2601.04500, 2026

    2026

  130. [138]

    Opeflo: Automated ux evaluation via simulated human web interaction with gui grounding.arXiv preprint arXiv:2604.09581, 2026

    Wee Joe Tan, Zi Rui Lucas Lim, Shashank Durgad, Karim Obegi, and Aiden Yiliu Li. Opeflo: Automated ux evaluation via simulated human web interaction with gui grounding.arXiv preprint arXiv:2604.09581, 2026

    2026 arXiv

  131. [139]

    Clawkeeper: Comprehensive safety protection for openclaw agents through skills, plugins, and watchers.arXiv preprint arXiv:2603.24414, 2026

    Songyang Liu, Chaozhuo Li, Chenxu Wang, Jinyu Hou, Zejian Chen, Litian Zhang, Zheng Liu, Qiwei Ye, Yiming Hei, Xi Zhang, et al. Clawkeeper: Comprehensive safety protection for openclaw agents through skills, plugins, and watchers.arXiv preprint arXiv:2603.24414, 2026

    2026

  132. [140]

    Vpi-bench: Visual prompt injection attacks for computer-use agents.arXiv preprint arXiv:2506.02456, 2025

    Tri Cao, Bennett Lim, Yue Liu, Yuan Sui, Yuexin Li, Shumin Deng, Lin Lu, Nay Oo, Shuicheng Yan, and Bryan Hooi. Vpi-bench: Visual prompt injection attacks for computer-use agents.arXiv preprint arXiv:2506.02456, 2025

    2025

  133. [141]

    Agentrae: Remote action execution through notification-based visual backdoors against screenshots-based mobile gui agents.arXiv preprint arXiv:2603.23007, 2026

    Yutao Luo, Haotian Zhu, Shuchao Pang, Zhigang Lu, Tian Dong, Yongbin Zhou, and Minhui Xue. Agentrae: Remote action execution through notification-based visual backdoors against screenshots-based mobile gui agents.arXiv preprint arXiv:2603.23007, 2026

    2026

  134. [142]

    Safepred: A predictive guardrail for computer-using agents via world models.arXiv preprint arXiv:2602.01725, 2026

    Yurun Chen, Zeyi Liao, Ping Yin, Taotao Xie, Keting Yin, and Shengyu Zhang. Safepred: A predictive guardrail for computer-using agents via world models.arXiv preprint arXiv:2602.01725, 2026

    2026

  135. [143]

    Permission manifests for web agents.arXiv preprint arXiv:2601.02371, 2025

    Samuele Marro, Alan Chan, Xinxing Ren, Lewis Hammond, Jesse Wright, Gurjyot Wanga, Tiziano Piccardi, Nuno Campos, Tobin South, Jialin Yu, et al. Permission manifests for web agents.arXiv preprint arXiv:2601.02371, 2025

    2025

  136. [144]

    Decepticon: How dark patterns manipulate web agents.arXiv preprint arXiv:2512.22894, 2025

    Phil Cuvin, Hao Zhu, and Diyi Yang. Decepticon: How dark patterns manipulate web agents.arXiv preprint arXiv:2512.22894, 2025

    2025

  137. [145]

    Slowba: An efficiency backdoor attack towards vlm-based gui agents.arXiv preprint arXiv:2603.08316, 2026

    Junxian Li, Tu Lan, Haozhen Tan, Yan Meng, and Haojin Zhu. Slowba: An efficiency backdoor attack towards vlm-based gui agents.arXiv preprint arXiv:2603.08316, 2026

    2026

  138. [146]

    Agentsentinel: An end-to-end and real-time security defense framework for computer- use agents

    Haitao Hu, Peng Chen, Yanpeng Zhao, and Yuqi Chen. Agentsentinel: An end-to-end and real-time security defense framework for computer- use agents. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, pages 3535–3549, 2025

    2025

  139. [147]

    Mip against agent: Malicious image patches hijacking multimodal os agents.arXiv preprint arXiv:2503.10809, 2025

    Lukas Aichberger, Alasdair Paren, Guohao Li, Philip Torr, Yarin Gal, and Adel Bibi. Mip against agent: Malicious image patches hijacking multimodal os agents.arXiv preprint arXiv:2503.10809, 2025

    2025

  140. [148]

    Websentinel: Detecting and localizing prompt injection attacks for web agents.arXiv preprint arXiv:2602.03792, 2026

    Xilong Wang, Yinuo Liu, Zhun Wang, Dawn Song, and Neil Gong. Websentinel: Detecting and localizing prompt injection attacks for web agents.arXiv preprint arXiv:2602.03792, 2026

    2026

  141. [149]

    Bradley Knox, and Kimin Lee

    Juyong Lee, Dongyoon Hahm, June Suk Choi, W. Bradley Knox, and Kimin Lee. MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control. InFortieth AAAI Conference on Artificial Intelligence, pages 37565–37573. AAAI Press, 2026

    2026

  142. [150]

    ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents

    Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, Nir Mashkif, and Segev Shlomov. ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents. InInternational Conference on Learning Representations, 2026. Poster

    2026

  143. [151]

    The blind spot of agent safety: How benign user instructions expose critical vulnerabilities in computer-use agents.arXiv preprint arXiv:2604.10577, 2026

    Xuwei Ding, Skylar Zhai, Linxin Song, Jiate Li, Taiwei Shi, Nicholas Meade, Siva Reddy, Jian Kang, and Jieyu Zhao. The blind spot of agent safety: How benign user instructions expose critical vulnerabilities in computer-use agents.arXiv preprint arXiv:2604.10577, 2026

    2026 arXiv

  144. [152]

    Safesearch: Automated red-teaming for the safety of llm-based search agents.arXiv preprint arXiv:2509.23694, 2025

    Jianshuo Dong, Sheng Guo, Hao Wang, Xun Chen, Zhuotao Liu, Tianwei Zhang, Ke Xu, Minlie Huang, and Han Qiu. Safesearch: Automated red-teaming for the safety of llm-based search agents.arXiv preprint arXiv:2509.23694, 2025

    2025

  145. [153]

    A systematization of security vulnerabilities in computer use agents.arXiv preprint arXiv:2507.05445, 2025

    Daniel Jones, Giorgio Severi, Martin Pouliot, Gary Lopez, Joris de Gruyter, Santiago Zanella-Beguelin, Justin Song, Blake Bull- winkel, Pamela Cortez, and Amanda Minnich. A systematization of security vulnerabilities in computer use agents.arXiv preprint arXiv:2507.05445, 2025

    2025

  146. [154]

    Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents.arXiv preprint arXiv:2410.02644, 2024

    Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents.arXiv preprint arXiv:2410.02644, 2024

    2024

  147. [155]

    Evil geniuses: Delving into the safety of llm-based agents.arXiv preprint arXiv:2311.11855, 2023

    Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. Evil geniuses: Delving into the safety of llm-based agents.arXiv preprint arXiv:2311.11855, 2023

    2023

  148. [156]

    Hidden ghost hand: Unveiling backdoor vulnerabilities in mllm-powered mobile gui agents.arXiv preprint arXiv:2505.14418, 2025

    Pengzhou Cheng, Haowen Hu, Zheng Wu, Zongru Wu, Tianjie Ju, Zhuosheng Zhang, and Gongshen Liu. Hidden ghost hand: Unveiling backdoor vulnerabilities in mllm-powered mobile gui agents.arXiv preprint arXiv:2505.14418, 2025

    2025

  149. [157]

    Badvla: Towards backdoor attacks on vision-language- action models via objective-decoupled optimization.arXiv preprint arXiv:2505.16640, 2025

    Xueyang Zhou, Guiyao Tie, Guowen Zhang, Hechang Wang, Pan Zhou, and Lichao Sun. Badvla: Towards backdoor attacks on vision-language- action models via objective-decoupled optimization.arXiv preprint arXiv:2505.16640, 2025

    2025

  150. [158]

    Measuring harmfulness of computer-using agents

    Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang, Ji Wang, Tianyu Shi, and Jiaxin Wen. Measuring harmfulness of computer-using agents. arXiv preprint arXiv:2508.00935, 2025

    2025

  151. [159]

    Secure and efficient access control for computer-use agents via context space

    Haochen Gong, Chenxiao Li, Rui Chang, and Wenbo Shen. Secure and efficient access control for computer-use agents via context space. arXiv preprint arXiv:2509.22256, 2025

    2025

  152. [160]

    cellmate: Sandboxing browser ai agents.arXiv preprint arXiv:2512.12594, 2025

    Luoxi Meng, Henry Feng, Ilia Shumailov, and Earlence Fernan- des. cellmate: Sandboxing browser ai agents.arXiv preprint arXiv:2512.12594, 2025

    2025

  153. [161]

    Redteamcua: Realistic adversarial testing of computer-use agents in hybrid web-os environments.arXiv preprint arXiv:2505.21936, 2025

    Zeyi Liao, Jaylen Jones, Linxi Jiang, Yuting Ning, Eric Fosler-Lussier, Yu Su, Zhiqiang Lin, and Huan Sun. Redteamcua: Realistic adversarial testing of computer-use agents in hybrid web-os environments.arXiv preprint arXiv:2505.21936, 2025

    2025

  154. [162]

    Hackworld: Evaluating computer-use agents on exploiting web application vulnerabilities.arXiv preprint arXiv:2510.12200, 2025

    Xiaoxue Ren, Penghao Jiang, Kaixin Li, Zhiyong Huang, Xiaoning Du, Jiaojiao Jiang, Zhenchang Xing, Jiamou Sun, and Terry Yue Zhuo. Hackworld: Evaluating computer-use agents on exploiting web application vulnerabilities.arXiv preprint arXiv:2510.12200, 2025

    2025

  155. [163]

    Injecagent: Benchmarking indirect prompt injections in tool-integrated large lan- guage model agents

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injections in tool-integrated large lan- guage model agents. InFindings of the Association for Computational Linguistics: ACL 2024, pages 10471–10506, 2024

    2024

  156. [164]

    Ghostei-bench: Do mobile agents resilience to environmental injection in dynamic on-device environments?arXiv preprint arXiv:2510.20333, 2025

    Chiyu Chen, Xinhao Song, Yunkai Chai, Yang Yao, Haodong Zhao, Lijun Li, Jie Li, Yan Teng, Gongshen Liu, and Yingchun Wang. Ghostei-bench: Do mobile agents resilience to environmental injection in dynamic on-device environments?arXiv preprint arXiv:2510.20333, 2025

    2025

  157. [165]

    Environmental injection attacks against gui agents in realistic dynamic environments.arXiv preprint arXiv:2509.11250, 2026

    Yitong Zhang, Ximo Li, Liyi Cai, and Jia Li. Environmental injection attacks against gui agents in realistic dynamic environments.arXiv preprint arXiv:2509.11250, 2026

    2026

  158. [166]

    Wasp: Benchmarking web agent security against prompt injection attacks.arXiv preprint arXiv:2504.18575, 2025

    Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. Wasp: Benchmarking web agent security against prompt injection attacks.arXiv preprint arXiv:2504.18575, 2025

    2025

  159. [167]

    EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage

    Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage. In International Conference on Learning Representations, 2025. Poster

    2025

  160. [168]

    Are gui agents focused enough? automated distraction via semantic-level ui element injection.arXiv preprint arXiv:2604.07831, 2026

    Wenkui Yang, Chao Jin, Haisu Zhu, Weilin Luo, Derek Yuen, Kun Shao, Huaibo Huang, Junxian Duan, Jie Cao, and Ran He. Are gui agents focused enough? automated distraction via semantic-level ui element injection.arXiv preprint arXiv:2604.07831, 2026

    2026 arXiv

  161. [169]

    It’s a trap! task-redirecting agent persuasion benchmark for web agents.arXiv preprint arXiv:2512.23128, 2025

    Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki, Will Howard, Lukas Aichberger, Chris Russell, Philip HS Torr, Adam Mahdi, Adel Bibi, et al. It’s a trap! task-redirecting agent persuasion benchmark for web agents.arXiv preprint arXiv:2512.23128, 2025

    2025

  162. [170]

    Investi- gating the impact of dark patterns on llm-based web agents.arXiv preprint arXiv:2510.18113, 2025

    Devin Ersoy, Brandon Lee, Ananth Shreekumar, Arjun Arunasalam, Muhammad Ibrahim, Antonio Bianchi, and Z Berkay Celik. Investi- gating the impact of dark patterns on llm-based web agents.arXiv preprint arXiv:2510.18113, 2025

    2025

  163. [171]

    Dual-modality multi-stage adversarial safety training: Robustifying multimodal web agents against cross-modal attacks.arXiv preprint arXiv:2603.04364, 2026

    Haoyu Liu, Dingcheng Li, Lukas Rutishauser, and Zeyu Zheng. Dual-modality multi-stage adversarial safety training: Robustifying multimodal web agents against cross-modal attacks.arXiv preprint arXiv:2603.04364, 2026

    2026

  164. [172]

    Agentdog: A diagnostic guardrail framework for ai agent safety and security.arXiv preprint arXiv:2601.18491, 2026

    Dongrui Liu, Qihan Ren, Chen Qian, Shuai Shao, Yuejin Xie, Yu Li, Zhonghao Yang, Haoyu Luo, Peng Wang, Qingyu Liu, et al. Agentdog: A diagnostic guardrail framework for ai agent safety and security.arXiv preprint arXiv:2601.18491, 2026

    2026 arXiv

  165. [173]

    Shieldagent: Shield- ing agents via verifiable safety policy reasoning.arXiv preprint arXiv:2503.22738, 2025

    Zhaorun Chen, Mintong Kang, and Bo Li. Shieldagent: Shield- ing agents via verifiable safety policy reasoning.arXiv preprint arXiv:2503.22738, 2025

    2025

  166. [174]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowl- edge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2024

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming llm agents via poisoning memory or knowl- edge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2024

    2024

  167. [175]

    Preference redirection via attention concentration: An attack on computer use agents.arXiv preprint arXiv:2604.08005, 2026

    Dominik Seip and Matthias Hein. Preference redirection via attention concentration: An attack on computer use agents.arXiv preprint arXiv:2604.08005, 2026

    2026 arXiv

  168. [176]

    When benign inputs lead to severe harms: Eliciting unsafe unintended behaviors of computer-use agents.arXiv preprint arXiv:2602.08235, 2026

    Jaylen Jones, Zhehao Zhang, Yuting Ning, Eric Fosler-Lussier, Pierre- Luc St-Charles, Yoshua Bengio, Dawn Song, Yu Su, and Huan Sun. When benign inputs lead to severe harms: Eliciting unsafe unintended behaviors of computer-use agents.arXiv preprint arXiv:2602.08235, 2026

    2026

  169. [177]

    Zero-permission manipulation: Can we trust large multimodal model powered gui agents?arXiv preprint arXiv:2601.12349, 2026

    Yi Qian, Kunwei Qian, Xingbang He, Ligeng Chen, Jikang Zhang, Tiantai Zhang, Haiyang Wei, Linzhang Wang, Hao Wu, and Bing Mao. Zero-permission manipulation: Can we trust large multimodal model powered gui agents?arXiv preprint arXiv:2601.12349, 2026

    2026

  170. [178]

    Hijacking jarvis: Benchmarking mobile gui agents against unprivileged third parties

    Guohong Liu, Jialei Ye, Jiacheng Liu, Yuanchun Li, Wei Liu, Pengzhi Gao, Jian Luan, and Yunxin Liu. Hijacking jarvis: Benchmarking mobile gui agents against unprivileged third parties. InProceedings of the 2nd International Workshop on Edge and Mobile Foundation Models, pages ...

    2025

  171. [179]

    Prompt infection: Llm-to-llm prompt injection within multi-agent systems.arXiv preprint arXiv:2410.07283, 2024

    Donghyun Lee and Mo Tiwari. Prompt infection: Llm-to-llm prompt injection within multi-agent systems.arXiv preprint arXiv:2410.07283, 2024

    2024

  172. [180]

    Multi-agent security tax: Trading off security and collaboration capa- bilities in multi-agent systems

    Pierre Peign ´e, Mikolaj Kniejski, Filip Sondej, Matthieu David, Jason Hoelscher-Obermaier, Christian Schroeder de Witt, and Esben Kran. Multi-agent security tax: Trading off security and collaboration capa- bilities in multi-agent systems. InProceedings of the AAAI Conference...

    2025

  173. [181]

    Agentdam: Pri- vacy leakage evaluation for autonomous web agents.arXiv preprint arXiv:2503.09780, 2025

    Arman Zharmagambetov, Chuan Guo, Ivan Evtimov, Maya Pavlova, Ruslan Salakhutdinov, and Kamalika Chaudhuri. Agentdam: Pri- vacy leakage evaluation for autonomous web agents.arXiv preprint arXiv:2503.09780, 2025

    2025

  174. [182]

    Privacy in action: Towards realistic privacy mitigation and evaluation for llm-powered agents.arXiv preprint arXiv:2509.17488, 2025

    Shouju Wang, Fenglin Yu, Xirui Liu, Xiaoting Qin, Jue Zhang, Qingwei Lin, Dongmei Zhang, and Saravan Rajmohan. Privacy in action: Towards realistic privacy mitigation and evaluation for llm-powered agents.arXiv preprint arXiv:2509.17488, 2025

    2025

  175. [183]

    Core: Reducing ui exposure in mobile agents via collaboration between cloud and local llms.arXiv preprint arXiv:2510.15455, 2025

    Gucongcong Fan, Chaoyue Niu, Chengfei Lyu, Fan Wu, and Guihai Chen. Core: Reducing ui exposure in mobile agents via collaboration between cloud and local llms.arXiv preprint arXiv:2510.15455, 2025

    2025

  176. [184]

    GUIGuard: Toward a General Framework for Privacy-Preserving GUI Agents, jan 2026

    Yanxi Wang, Zhiling Zhang, Wenbo Zhou, Weiming Zhang, Jie Zhang, Qiannan Zhu, Yu Shi, Shuxin Zheng, and Jiyan He. GUIGuard: Toward a General Framework for Privacy-Preserving GUI Agents, jan 2026. arXiv preprint arXiv:2601.18842

    2026

  177. [185]

    Veriguard: Enhancing llm agent safety via verified code generation.arXiv preprint arXiv:2510.05156, 2025

    Lesly Miculicich, Mihir Parmar, Hamid Palangi, Krishnamurthy Dj Dvijotham, Mirko Montanari, Tomas Pfister, and Long T Le. Veriguard: Enhancing llm agent safety via verified code generation.arXiv preprint arXiv:2510.05156, 2025

    2025

  178. [186]

    Mitigating indirect prompt injection via instruction- following intent analysis.arXiv preprint arXiv:2512.00966, 2025

    Mintong Kang, Chong Xiang, Sanjay Kariyappa, Chaowei Xiao, Bo Li, and Edward Suh. Mitigating indirect prompt injection via instruction- following intent analysis.arXiv preprint arXiv:2512.00966, 2025

    2025

  179. [187]

    When actions go off-task: Detecting and correcting misaligned actions in computer-use agents

    Yuting Ning, Jaylen Jones, Zhehao Zhang, Chentao Ye, Weitong Ruan, Junyi Li, Rahul Gupta, and Huan Sun. When actions go off-task: Detecting and correcting misaligned actions in computer-use agents. arXiv preprint arXiv:2602.08995, 2026

    2026

  180. [188]

    Mirrorguard: Toward secure computer-use agents via simulation-to-real reasoning correction.arXiv preprint arXiv:2601.12822, 2026

    Wenqi Zhang, Yulin Shen, Changyue Jiang, Jiarun Dai, Geng Hong, and Xudong Pan. Mirrorguard: Toward secure computer-use agents via simulation-to-real reasoning correction.arXiv preprint arXiv:2601.12822, 2026

    2026

  181. [189]

    VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification

    Jungjae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, and Insik Shin. VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification. InProceedings of the 31st Annual International Conference on Mobile Computing...

    2025

  182. [190]

    Gem: Gaussian embedding modeling for out-of- distribution detection in gui agents

    Zheng Wu, Pengzhou Cheng, Zongru Wu, Lingzhong Dong, and Zhuosheng Zhang. Gem: Gaussian embedding modeling for out-of- distribution detection in gui agents. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33989–33997, 2026

    2026

  183. [191]

    Browsesafe: Understanding and preventing prompt injection within ai browser agents.arXiv preprint arXiv:2511.20597, 2025

    Kaiyuan Zhang, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, and Ninghui Li. Browsesafe: Understanding and preventing prompt injection within ai browser agents.arXiv preprint arXiv:2511.20597, 2025

    2025

  184. [192]

    Propen- sitybench: Evaluating latent safety risks in large language models via an agentic approach.arXiv preprint arXiv:2511.20703, 2025

    Udari Madhushani Sehwag, Shayan Shabihi, Alex McAvoy, Vikash Sehwag, Yuancheng Xu, Dalton Towers, and Furong Huang. Propen- sitybench: Evaluating latent safety risks in large language models via an agentic approach.arXiv preprint arXiv:2511.20703, 2025

    2025

Pith tools

Reviewed May 11, 2026 · model on record in the stance chip above.