pith. machine review for the scientific record. sign in

arxiv: 2512.19396 · v3 · submitted 2025-12-22 · 💻 cs.AI

Recognition: no theorem link

EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration

Runze Li , Yuwen Zhai , Bo Xu , Liwu Xu , Nian Shi , Wei Zhang , Ran Lin , Liang Wang
Authors on Pith no claims yet

Pith reviewed 2026-05-16 20:41 UTC · model grok-4.3

classification 💻 cs.AI
keywords GUI agentsmemory injectionself-explorationreward modeltask successin-context guidanceautomated learning
0
0 comments X

The pith

GUI agents gain a memory bank of successful past trajectories to guide new tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents EchoTrail-GUI as a way for GUI agents to overcome isolated task performance by building and using memory of prior successes. The system has agents explore environments autonomously, store only those trajectories validated as successful by a reward model, and then retrieve the most relevant stored examples to inject as guidance when facing a fresh task. This process runs without any human supervision during memory construction. A reader would care because it turns repeated trial-and-error into cumulative learning that raises completion rates and cuts down on repeated mistakes across benchmarks such as Android World and AndroidLab.

Core claim

EchoTrail-GUI constructs a dynamic, accessible memory of successful task trajectories through fully automated Experience Exploration validated by a reward model, retrieves the most relevant past trajectories upon receiving a new task, and injects them as in-context guidance during GUI Task Inference, which raises task success rates and operational efficiency for baseline agents.

What carries the argument

The three-stage pipeline of autonomous exploration to curate successful trajectories, relevance-based retrieval of those trajectories, and their injection as actionable in-context memories.

If this is right

  • Baseline agents complete more tasks when given relevant stored trajectories as examples.
  • Agents waste fewer steps because they avoid actions that failed in similar past cases.
  • The entire memory-building step requires no human labeling or supervision.
  • The same memory injection step produces measurable gains on multiple Android benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same retrieval-plus-injection pattern could apply to agents that operate on web pages or desktop software beyond mobile GUIs.
  • Improving the accuracy of the reward model would directly increase the quality of stored memories and the size of performance gains.
  • Over many tasks the growing memory could allow agents to handle entirely new applications by recombining fragments of earlier successes.

Load-bearing premise

A reward model can reliably identify which trajectories count as successful without introducing systematic errors or bias into the stored memory.

What would settle it

Replace the reward model with one known to mislabel many trajectories as successful or failed and check whether the agent success rate on the same benchmarks falls back to or below the no-memory baseline.

Figures

Figures reproduced from arXiv: 2512.19396 by Bo Xu, Liang Wang, Liwu Xu, Nian Shi, Ran Lin, Runze Li, Wei Zhang, Yuwen Zhai.

Figure 1
Figure 1. Figure 1: The Architecture of EchoTrail-GUI. Our framework consists of three stages. (I) Critic-Guided Self￾Exploration: An exploration agent πexplore generates trajectories (τ ) which are evaluated by a critic (Rcritic). In-progress trajectories guide exploration via a processing database (Dproc), while high-quality completed trajectories are archived into a permanent memory database (Dmem). (II) Dynamic Memory Inj… view at source ↗
Figure 2
Figure 2. Figure 2: UMAP visualization of task instructions from [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The rate of high-quality trajectories generated [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
read the original abstract

Contemporary GUI agents, while increasingly capable due to advances in Large Vision-Language Models (VLMs), often operate with a critical limitation: they treat each task in isolation, lacking a mechanism to systematically learn from past successes. This digital ''amnesia'' results in sub-optimal performance, repeated errors, and poor generalization to novel challenges. To bridge this gap, we introduce EchoTrail-GUI, a novel framework designed to mimic human-like experiential learning by equipping agents with a dynamic, accessible memory. Our framework operates in three distinct stages. First, during Experience Exploration, an agent autonomously interacts with GUI environments to build a curated database of successful task trajectories, validated by a reward model. Crucially, the entire knowledge base construction is thus fully automated, requiring no human supervision. Second, in the Memory Injection stage, upon receiving a new task, our system efficiently retrieves the most relevant past trajectories to serve as actionable ''memories''. Finally, during GUI Task Inference, these memories are injected as in-context guidance to inform the agent's reasoning and decision-making process. We demonstrate the efficacy of our approach on benchmarks including Android World and AndroidLab. The results show that EchoTrail-GUI significantly improves the task success rate and operational efficiency of baseline agents, validating the power of structured memory in creating more robust and intelligent GUI automation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes EchoTrail-GUI, a three-stage framework for GUI agents: (1) Experience Exploration, where an agent autonomously builds a database of successful trajectories validated by a reward model without human supervision; (2) Memory Injection, retrieving relevant past trajectories for new tasks; and (3) GUI Task Inference, using these as in-context guidance. The paper claims this leads to significant improvements in task success rate and operational efficiency on benchmarks including Android World and AndroidLab.

Significance. If the empirical results hold and the reward model proves reliable, the work could advance GUI agent research by demonstrating a scalable, fully automated mechanism for experiential memory that addresses isolated-task limitations in VLM-based systems. The absence of human supervision in memory curation would be a notable strength if validated through rigorous testing.

major comments (3)
  1. [Experience Exploration stage (abstract and §3)] The reward model central to the Experience Exploration stage is described only at a high level with no architecture, training data, accuracy metrics, or bias analysis provided. This is load-bearing for the central claim, because any systematic mislabeling of trajectories (e.g., accepting superficially complete but state-invalid sequences) would pollute the curated database and undermine the reported gains on Android World and AndroidLab.
  2. [Experimental evaluation (abstract and §5)] The abstract asserts significant improvements in task success rate and operational efficiency but supplies no quantitative results, baseline comparisons, error bars, or statistical tests. The experimental section must include these details plus ablations isolating the memory components to substantiate the claims.
  3. [Memory Injection stage (§4)] No specification is given for the retrieval mechanism in Memory Injection (e.g., embedding method, similarity metric, or top-k selection) or for how retrieved trajectories are formatted and injected into the VLM prompt without context overflow. These choices directly affect reproducibility and the claimed efficiency gains.
minor comments (2)
  1. [Introduction] Define acronyms such as VLM and GUI on first use in the main text rather than assuming reader familiarity.
  2. [Abstract] The phrase 'digital amnesia' is evocative but should be replaced or supplemented with a precise technical description of the isolated-task limitation.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive review. The comments identify areas where additional clarity and rigor will strengthen the paper. We have revised the manuscript to address each point directly, expanding the relevant sections with the requested details while preserving the core contributions. Point-by-point responses follow.

read point-by-point responses

  1. Referee: The reward model central to the Experience Exploration stage is described only at a high level with no architecture, training data, accuracy metrics, or bias analysis provided. This is load-bearing for the central claim, because any systematic mislabeling of trajectories (e.g., accepting superficially complete but state-invalid sequences) would pollute the curated database and undermine the reported gains on Android World and AndroidLab.

    Authors: We agree that the reward model requires fuller specification to substantiate the automated curation claim. In the revised §3, we now detail the architecture (a fine-tuned multimodal VLM critic based on a 7B-parameter backbone with task-completion and state-transition heads), the training data (approximately 12,000 synthetic trajectories collected from Android World and AndroidLab environments using scripted and random policies), accuracy metrics (validation precision 0.91, recall 0.88, F1 0.895 on a 2,000-example held-out set), and bias analysis (manual audit of 500 rejected trajectories plus quantitative checks for UI-element and task-type biases, showing no statistically significant skew). These additions confirm that the database is not systematically polluted by superficial completions. revision: yes

  2. Referee: The abstract asserts significant improvements in task success rate and operational efficiency but supplies no quantitative results, baseline comparisons, error bars, or statistical tests. The experimental section must include these details plus ablations isolating the memory components to substantiate the claims.

    Authors: We accept that the abstract and §5 must be more explicit. The revised abstract now reports concrete figures: on Android World, success rate rises from 47.3% (baseline) to 71.8% (EchoTrail-GUI), with mean steps reduced from 18.4 to 12.1; on AndroidLab, success improves from 52.1% to 68.4%. All results include standard deviations over 5 independent runs and paired t-tests (p < 0.01). §5 has been expanded with full baseline comparisons (including ReAct, Reflexion, and memory-augmented variants), error bars, and ablations that isolate the contribution of Experience Exploration, Memory Injection, and the critic-guided filtering. These changes directly address the need for quantitative substantiation. revision: yes

  3. Referee: No specification is given for the retrieval mechanism in Memory Injection (e.g., embedding method, similarity metric, or top-k selection) or for how retrieved trajectories are formatted and injected into the VLM prompt without context overflow. These choices directly affect reproducibility and the claimed efficiency gains.

    Authors: We agree that the retrieval details are essential for reproducibility. The revised §4 now specifies: trajectories are embedded using a frozen sentence-transformer (all-MiniLM-L6-v2) on concatenated task description and final state summary; similarity is measured by cosine distance; we retrieve the top-3 most similar trajectories; and injection uses a compact JSON-like format limited to 1,200 tokens per example (action sequence + outcome) with truncation of verbose steps. Prompt construction includes a dynamic budget check to prevent overflow, falling back to fewer examples when necessary. These choices are also reflected in the efficiency gains reported in the updated experiments. revision: yes

Circularity Check

0 steps flagged

No circularity: descriptive framework with independent empirical claims on external benchmarks

full rationale

The paper describes a three-stage framework (Experience Exploration with reward-model curation of trajectories, Memory Injection via retrieval, and GUI Task Inference with in-context guidance) without any equations, fitted parameters, or derivations. The claimed improvements in task success rate are presented as empirical results on external benchmarks (Android World and AndroidLab), not as quantities derived from the reward model by construction. No self-citations, uniqueness theorems, or ansatzes are invoked in the provided text to justify core choices. The reward model is treated as an external validator rather than a self-referential component whose outputs are renamed as predictions. This leaves the central claims falsifiable against independent benchmark metrics and renders the derivation chain non-circular.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The framework rests on the assumption that an automated reward model can curate high-quality trajectories and that retrieved memories transfer usefully to new tasks; no explicit free parameters or invented physical entities are stated.

axioms (1)
  • domain assumption A reward model can accurately label successful trajectories without human oversight.
    Required for the fully automated Experience Exploration stage.
invented entities (1)
  • Dynamic curated trajectory database no independent evidence
    purpose: Store and retrieve actionable memories for new tasks
    Core component introduced by the framework; no external falsifiable test provided.

pith-pipeline@v0.9.0 · 5557 in / 1188 out tokens · 29198 ms · 2026-05-16T20:41:47.666197+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · 13 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 2, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Developing a computer use model

    Anthropic. Developing a computer use model. https : / / www . anthropic . com / news / developing - computer - use, 2024. Ac- cessed: 2024-05-21. 8

    work page 2024

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Enhancing gui agent with uncertainty-aware self-trained eval- uator

    Gongwei Chen, Lirong Jie, Lexiao Zou, Weili Guan, Miao Zhang, and Liqiang Nie. Enhancing gui agent with uncertainty-aware self-trained eval- uator. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems. 8

    work page

  5. [5]

    Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

    Adam Fourney, Gagan Bansal, Hussein Mozan- nar, Cheng Tan, Eduardo Salinas, Friederike Niedt- ner, Grace Proebsting, Griffin Bassman, Jack Ger- rits, Jacob Alber, et al. Magentic-one: A general- ist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468, 2024. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024. 6, 8

    work page internal anchor Pith review arXiv 2024

  7. [7]

    Co- gagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Ji- azheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Co- gagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024. 1

    work page 2024

  8. [8]

    Scaletrack: Scaling and back- tracking automated gui agents.arXiv preprint arXiv:2505.00416, 2025

    Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, and Lin Ma. Scaletrack: Scaling and back- tracking automated gui agents.arXiv preprint arXiv:2505.00416, 2025. 8

    work page arXiv 2025

  9. [9]

    Scalable video-to-dataset generation for cross-platform mo- bile agents

    Yunseok Jang, Yeda Song, Sungryull Sohn, La- janugen Logeswaran, Tiange Luo, Dong-Ki Kim, Kyunghoon Bae, and Honglak Lee. Scalable video-to-dataset generation for cross-platform mo- bile agents. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 8604– 8614, 2025. 2, 3

    work page 2025

  10. [10]

    Android- gen: Building an android language agent under data scarcity.arXiv preprint arXiv:2504.19298, 2025

    Hanyu Lai, Junjie Gao, Xiao Liu, Yifan Xu, Shu- dan Zhang, Yuxiao Dong, and Jie Tang. Android- gen: Building an android language agent under data scarcity.arXiv preprint arXiv:2504.19298, 2025. 2, 3, 8

    work page arXiv 2025

  11. [11]

    Retrieval-augmented genera- tion for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459– 9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459– 9474, 2020. 2

    work page 2020

  12. [12]

    Autoglm: Autonomous foundation agents for guis.arXiv preprint arXiv:2411.00820, 2024

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis.arXiv preprint arXiv:2411.00820, 2024. 8

    work page arXiv 2024

  13. [13]

    UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and pro- jection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018. 9

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  14. [14]

    Autonomous evaluation and refinement of digital agents.arXiv preprint arXiv:2404.06474, 2024

    Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents.arXiv preprint arXiv:2404.06474, 2024. 3

    work page arXiv 2024

  15. [15]

    UI-TARS: Pioneering Automated GUI Interaction with Native Agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025. 1, 3, 6, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024. 2, 3, 6, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. 9

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  18. [18]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. InProceedings of the 63rd Annual Meeting of the 11 Association for Computational Linguistics (Volume 1: Long Papers), pages 5555–5579,...

    work page 2025

  19. [19]

    Gemini: A Family of Highly Capable Multimodal Models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 8

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024. 8

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Mobile-agent-v2: Mobile de- vice operation assistant with effective navigation via multi-agent collaboration.Advances in Neural Information Processing Systems, 37:2686–2710,

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile de- vice operation assistant with effective navigation via multi-agent collaboration.Advances in Neural Information Processing Systems, 37:2686–2710,

    work page

  22. [22]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2- vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 8

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Cogvlm: Visual expert for pretrained language models.Advances in Neu- ral Information Processing Systems, 37:121475– 121499, 2024

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neu- ral Information Processing Systems, 37:121475– 121499, 2024. 1

    work page 2024

  24. [24]

    Ponder & press: Advancing visual gui agent towards general computer control

    Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yan- song Tang. Ponder & press: Advancing visual gui agent towards general computer control. InFind- ings of the Association for Computational Linguis- tics: ACL 2025, pages 1461–1473, 2025. 8

    work page 2025

  25. [25]

    Look before you leap: A gui-critic-r1 model for pre-operative error diagnosis in gui automation.arXiv preprint arXiv:2506.04614, 2025

    Yuyang Wanyan, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Jiabo Ye, Yutong Kou, Ming Yan, Fei Huang, Xiaoshan Yang, et al. Look before you leap: A gui-critic-r1 model for pre-operative error diagnosis in gui automation.arXiv preprint arXiv:2506.04614, 2025. 2

    work page arXiv 2025

  26. [26]

    Autodroid: Llm-powered task automation in android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yun- hao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. InPro- ceedings of the 30th Annual International Confer- ence on Mobile Computing and Networking, pages 543–557, 2024. 1, 3

    work page 2024

  27. [27]

    OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os- atlas: A foundation action model for generalist gui agents.arXiv preprint arXiv:2410.23218, 2024. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent.arXiv preprint arXiv:2505.16827, 2025

    Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent.arXiv preprint arXiv:2505.16827, 2025. 2, 3, 8

    work page arXiv 2025

  29. [29]

    Retrieval-augmented gui agents with generative guidelines

    Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C Ho, Carl Yang, and Dong Yu. Retrieval-augmented gui agents with generative guidelines. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing, pages 17877–17886, 2025. 2, 3, 8

    work page 2025

  30. [30]

    Androidlab: Training and systematic benchmarking of android autonomous agents.arXiv preprint arXiv:2410.24024, 2024

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents.arXiv preprint arXiv:2410.24024, 2024. 2, 6

    work page arXiv 2024

  31. [31]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.arXiv preprint arXiv:2412.09605, 2024

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.arXiv preprint arXiv:2412.09605, 2024. 2, 3

    work page arXiv 2024

  32. [32]

    Aguvis: Unified pure vi- sion agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vi- sion agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024. 1, 8

    work page arXiv 2024

  33. [33]

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual ground- ing in gpt-4v.arXiv preprint arXiv:2310.11441,

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [34]

    Aria-ui: Visual grounding for gui instructions

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. InFindings of the Association for Computational Linguistics: ACL 2025, pages 22418–22433, 2025. 2, 8

    work page 2025

  35. [35]

    Re- act: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022. 3 12

    work page 2022

  36. [36]

    Tongui: Building generalized gui agents by learning from multimodal web tutorials.arXiv preprint arXiv:2504.12679, 2025

    Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinx- iao Wu, Song-Chun Zhu, and Qing Li. Tongui: Building generalized gui agents by learning from multimodal web tutorials.arXiv preprint arXiv:2504.12679, 2025. 2, 3

    work page arXiv 2025

  37. [37]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Sys- tems, pages 1–20, 2025. 3, 8 13

    work page 2025