Pith. sign in

REVIEW 5 major objections 5 minor 49 references

A transition-graph map of an app turns exploration into 14.8-step mobile agent training trajectories, and fine-tuning on them lifts task success from 62.6% to 77.3%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:14 UTC pith:JIR37UKU

load-bearing objection Useful synthesis pipeline and dataset for long-horizon GUI training, with real transfer evidence; main caveats are unseen artifacts and unvalidated executability. the 5 major comments →

arxiv 2607.18046 v1 pith:JIR37UKU submitted 2026-07-20 cs.LG

SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

Zhuohang Fan , Beichen Zhang , Yuanfa Li , Changqiao Wu , Wei Liu , Jian Luan , Weigang Zhang This is my paper
classification cs.LG
keywords GUI agentsdata synthesistrajectory generationUI transition graphmobile appsvision-language modelslong-horizon tasksfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that long-horizon GUI-agent training data can be synthesized without human demonstrations by first exploring an app to build an explicit graph of screens, elements, and the transitions between them, then planning multi-step tasks over that graph. It reports that the resulting dataset, SEE, contains trajectories averaging 14.8 steps from element-dense screens (24.63 elements per screen), and that fine-tuning a 4B vision-language agent on this data raises success rate on held-out SEE tasks from 62.61% to 77.29% while also improving transfer to AndroidControl high-level tasks. A sympathetic reader would care because this attacks the data bottleneck that currently limits GUI agents in real, crowded, frequently-updated mobile apps, and because the graph decouples expensive exploration from cheap synthesis.

Core claim

SEE's central claim is that an explicit screen–element transition graph, built once per app through structure-aware exploration with reflection-based edge verification, is a sufficient substrate for synthesizing executable long-horizon trajectories. Task construction samples target screens, an LLM orders them into subgoals, and graph search (e.g., BFS) concatenates verified transitions into an action sequence; low-level step instructions are rendered from stored node and edge semantics rather than by re-analyzing screenshots at every step. The paper reports that this yields 47K training trajectories with an average length of 14.8 steps and 24.63 elements per screen, and that fine-tuning on t

What carries the argument

The load-bearing object is the bipartite UI transition graph G=(V,E). V contains screen nodes (abstract UI states) and element nodes (semantic descriptions of UI elements); E contains transition edges (action-induced screen-to-screen moves, each carrying a refined textual explanation) and containment edges (which elements appear on which screen). Exploration builds this graph with a scoring function that combines semantic alignment, layout priors, and history penalties, and a reflection step verifies each transition before it is added. Synthesis then treats the graph as a ground-truth model of the app: subgoal sequences are mapped to paths via BFS, and node/edge semantics are reused to rende

Load-bearing premise

The graph built during exploration is a faithful model of the live apps—specifically, that a path of individually verified edges, when concatenated, corresponds to a sequence a user could execute to actually complete the high-level instruction in the real interface.

What would settle it

Re-run a random sample of SEE's synthesized trajectories in the actual apps (or a high-fidelity emulator), without the simulator's oracle actions, and check whether the terminal screen matches the high-level instruction's intended result; if a substantial fraction of trajectories fail despite each edge passing LLM reflection, the graph-based synthesis chain is not executable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents fine-tuned on SEE improve success rate on held-out SEE-Test from 62.61% to 77.29% (Qwen3-VL 4B), with grounding accuracy rising from 60.79% to 69.91%, showing dense screens are the main bottleneck and that graph-synthesized data addresses it.
  • Training on SEE transfers to an external benchmark: on AndroidControl high-level tasks, Qwen3-VL 4B success rate improves from 60.1 to 67.8, indicating reusable procedural knowledge beyond training apps.
  • SEE produces trajectories with a 14.8-step average, substantially longer than existing public GUI datasets, and 24.63 elements per screen, making it a harder benchmark that exposes grounding failures.
  • Reflection-based graph refinement cuts the incorrect-edge rate from about 27–30% to about 9–11% at both evaluation scales, supporting the claim that verified edges make synthesized trajectories more reliable.
  • The screen-node matching module maintains 96.9% classification accuracy at the largest graph scale, versus 58–75% for global matching baselines, suggesting the abstraction function keeps the graph compact and accurate as it grows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that the same graph can be reused to synthesize arbitrarily many trajectories and to adjust difficulty by subgoal distance; the paper only sketches this controllability, but it implies a data flywheel where one exploration pass funds many training episodes.
  • The paper's success metric on SEE-Test is execution-based in a simulated environment; a natural extension would be re-executing synthesized full trajectories in the live apps to verify that a path that is semantically consistent at each edge actually completes the high-level instruction, something the paper does not report.
  • The reflection step depends on the LLM's ability to judge whether a screen is 'consistent' with the intended action; if that judgment is noisy for visually crowded screens, errors in edge verification propagate directly into every trajectory using that edge, so the method's ceiling is tied to the verifier's accuracy.
  • The transfer gains on AndroidControl concentrate in high-level tasks and action-type accuracy, implying the subgoal-aligned supervision, not grounding on specific layouts, is what generalizes; a testable prediction is that adding a low-level-only variant of SEE would lose most of the cross-benchmark benefit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SEE, a two-stage framework for synthesizing long-horizon mobile GUI agent trajectories. In the exploration stage, SEE parses screens, uses a VLM to describe elements, selects actions via a scoring function, and constructs a screen–element transition graph with reflection-based edge verification. In the synthesis stage, SEE samples subgoals from screen nodes, plans paths via BFS, and generates high-level instructions, ordered subgoal lists, and step-level action descriptions from graph semantics. The resulting SEE dataset has 47k training trajectories with an average length of 14.8 steps and 24.63 elements per screen. Fine-tuning Qwen3-VL 4B on SEE-Train raises SEE-Test success rate from 62.61% to 77.29%, and on AndroidControl-High from 60.1% to 67.8%. The paper also reports graph-quality improvements from reflection and exploration-efficiency gains. The authors state they will release code and dataset.

Significance. If the central claims hold, SEE is a useful contribution: it combines structure acquisition with graph-based trajectory composition, avoids manual annotation, targets the long-horizon, element-dense regime that existing datasets under-serve, and provides multi-level supervision. The external AndroidControl transfer experiment is a genuine out-of-distribution check and is one of the strongest pieces of evidence. The dataset statistics suggest a real gap in trajectory length and page complexity. However, the report identifies load-bearing weaknesses in the screen-state abstraction, the executable-trajectory guarantee, and the edge-filtering consistency, which currently prevent the results from being fully convincing.

major comments (5)
  1. [§3.2.2, §3.4.2, §5.5] The screen-state abstraction identifies nodes using only elements from predefined regions of interest R (top/bottom bars), with equivalence decided by Eq. (4) and an unspecified threshold. A BFS path found on this graph may concatenate edges recorded from different concrete instances of the same node, and the first edge of a plan need not be executable from another instance with the same bars but different main content. The paper never executes a synthesized full trajectory back in the live app; §5.5 only audits edges with an LLM. This directly affects the abstract's claim of 'executable trajectories' and the value of fine-tuning on SEE. Please provide a concrete validation protocol—e.g., executing a sample of synthesized trajectories in the original apps, or at minimum demonstrating that paths from arbitrary node instances are executable—and discuss how the abstraction granularity affec
  2. [§3.3 vs §5.5] There is an explicit contradiction: §3.3 states 'Only the verified transition edge will be added to the transition graph G,' while §5.5 states 'During exploration, all discovered transitions are added into the graph, while the reflection module only attaches review labels and does not modify edges online.' If the deployed pipeline filters edges, then the graph-quality study of §5.5, evaluated on the unfiltered graph, does not characterize the actual synthesis graph. If it does not filter, then §3.3 is inaccurate and the synthesized trajectories may contain invalid edges. This must be resolved because the reflection mechanism is a stated contribution for 'improving transition reliability.'
  3. [§5.1, Tables 4–6] No repeated runs, error bars, or significance tests are reported. Several transfer gains are small (e.g., UI-Genie 3B on AndroidControl-High SR 72.9→73.7; Qwen2.5-VL 7B Low SR 91.4→91.9, with Grounding decreasing from 94.7 to 90.8). Without multiple random seeds or statistical testing, the central fine-tuning claim cannot be distinguished from noise. Please report means/standard deviations over at least 3 runs for the main tables, and state the number of evaluation episodes. This is a standard expectation for fine-tuning comparisons in this area.
  4. [Eq. (4), §3.2.2] The screen-equivalence criterion is not well specified. The text says 'If a sufficient number of aligned pairs have low matching cost,' but neither the threshold on matching cost nor the required number of aligned pairs is given. Since this abstraction is what makes BFS paths reusable, the missing specification is a reproducibility and correctness gap. Please provide the exact threshold, the values of α and β, and ideally a sensitivity analysis showing how the graph size and downstream success change with the threshold.
  5. [§5.5, Table 7] The graph-quality evaluation uses a 'separate multimodal LLM auditor' that is likely from the same model family (Qwen) as the exploration LLM, and the paper reports no agreement analysis or human validation on the auditor's judgments. More importantly, the metric is edge-level 'semantic and functional validity' according to the auditor, not actual task completion. Given that the reflection step is one of the main differences from prior pipelines, please provide a more direct evaluation—e.g., human annotation of a sample of accepted/rejected edges, or execution of a subset of accepted paths—and report inter-annotator agreement if feasible.
minor comments (5)
  1. [General] The paper retains template artifacts: 'Conference’17, July 2017' in the header and 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' at the end. These should be cleaned.
  2. [Table 1] The '×' in 'SEE-Train47K×14.8' is unclear; it seems to indicate approximate multiplication of size and length, but it should be stated explicitly (e.g., '47K trajectories, average length 14.8').
  3. [Table 6] Some rows report '–' for Type/Grounding on AndroidControl; please state why these metrics are unavailable for those models/settings, or provide the missing values.
  4. [§4] The coverage percentages in Figure 4 are described as multi-label, but the presentation is hard to parse from the text alone; a short explanation of the normalization would help.
  5. [§3.4.3] The paper refers to 'supplementary material' for prompts and additional results, but no supplementary file is included in the arXiv submission. The authors should ensure these materials are available at submission time.

Circularity Check

0 steps flagged

No significant circularity: SEE's derivation is self-contained and its central claim is grounded by an external benchmark (AndroidControl), not by its own graph or fitted parameters.

full rationale

No step in SEE's derivation chain equates its output to its input by construction. The exploration stage (§3.2) builds the transition graph from observed app interactions; the synthesis stage (§3.4) plans paths over that graph and renders instructions from stored node/edge semantics; the downstream evaluations measure agent behavior on SEE-Test and on the independent AndroidControl benchmark (§5.4). There are no fitted parameters later renamed as predictions. The screen-equivalence rule in Eq. 4 is a perceptual matching criterion, not a target quantity derived from the same rule, and the graph-quality study (§5.5) uses human-annotated node labels plus an LLM auditor rather than a circular reuse of the generation output. Two caveats are correctness issues, not circularity. First, §3.3 says 'Only the verified transition edge will be added to the transition graph G,' while §5.5 states 'all discovered transitions are added into the graph, while the reflection module only attaches review labels and does not modify edges online'; this internal inconsistency weakens the 'executable trajectories' claim but does not make the derivation circular. Second, the 'independent LLM auditor' is from the same model family as the generator, so the audit is not fully independent. These weaken the evidence for executability, but the AndroidControl transfer results provide external, non-circular grounding for the claim that SEE supervision improves agents. No load-bearing self-citation or uniqueness-imported-from-authors pattern is present.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical or architectural entities; the transition graph and dataset are artifacts of the method. The main unspoken costs are the free threshold/hyperparameter choices listed above and the reliance on VLM quality for edge and instruction correctness.

free parameters (3)
  • alpha, beta in screen-state matching cost (Eq. 4) = not specified
    Weights for visual embedding vs. positional consistency when deciding whether two observations are the same screen; no values or tuning procedure are given.
  • screen-state equivalence threshold = not specified
    Section 3.2.2 says 'a sufficient number of aligned pairs have low matching cost' but never defines the threshold; this controls node granularity and therefore all downstream graph paths.
  • exploration hyperparameters (top-K, random probability, back-trigger rule) = not specified
    Top-K candidate elements and random exploration probability in Section 3.2.1, and the Back rule in Section 3.1, are described qualitatively; their values affect coverage and graph shape.
axioms (3)
  • domain assumption Observations can be faithfully abstracted into discrete UI states via visual matching (Eq. 4) and Hungarian alignment.
    Graph nodes depend on this abstraction; if state deduplication fails, planned paths are wrong. Invoked in Section 3.2.2.
  • domain assumption VLM semantic descriptions and reflection labels are accurate enough to validate edges and render instructions.
    The graph and synthesized low-level instructions inherit any VLM errors; no human verification is performed on full synthesized trajectories. Invoked in Sections 3.3 and 3.4.3.
  • domain assumption BFS paths over verified edges correspond to executable actions in the live app.
    Synthesis concatenates graph edges without re-executing or confirming the full trajectory; assumes no environmental drift or unmodeled dependencies. Invoked in Section 3.4.2.

pith-pipeline@v1.3.0-alltime-deepseek · 14736 in / 11123 out tokens · 112087 ms · 2026-08-01T16:14:12.394957+00:00 · methodology

0 comments
read the original abstract

Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps. Existing pipelines often rely on costly human demonstrations or on-policy framework, which tends to over-sample common flows while missing rare transitions and complex multi-step procedures. To address this problem, we propose SEE, a two-stage data synthesis framework consisting of (i) an efficient exploration stage that builds an explicit UI transition graph over screens and elements, and (ii) a graph-based synthesis stage that composes diverse multi-step trajectories via planning and controlled sampling. This design yields reproducible and explainable data generation, while explicitly preventing spurious cycles and enabling long-horizon composition. Across multiple real-world apps, SEE produces trajectories with an average length of 14.8 steps while avoiding spurious loops, and agents fine-tuned on SEE achieve improved task success and generalization to unseen screens. We will publicly release our synthesis code and dataset.

Figures

Figures reproduced from arXiv: 2607.18046 by Beichen Zhang, Changqiao Wu, Jian Luan, Weigang Zhang, Wei Liu, Yuanfa Li, Zhuohang Fan.

Figure 1
Figure 1. Figure 1: SEE targets the long-horizon, high-complexity regime of real-world mobile apps. Left: our dataset is built from popular, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SEE. In the exploration stage, the agent interacts with the app to build a screen–element transition graph. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Vision-and-semantics guided exploration for transition graph construction. Candidate UI elements are parsed and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Dataset coverage and instruction composition. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 11 linked inside Pith

  1. [1]

    Anthropic. 2024. Developing a computer use model. https://www.anthropic. com/news/developing-computer-use

    2024

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    Pith/arXiv arXiv 2023

  3. [3]

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yin- heng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Ke- unho Jang, and Zheng Hui. 2025. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. InProceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learni...

    2025

  4. [4]

    Card, Thomas P

    Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The keystroke-level model for user performance time with interactive systems.Commun. ACM23, 7 (July 1980), 396–410

    1980

  5. [5]

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Guozhi Wang, Dingyu Zhang, Shuai Ren, and Hongsheng Li. 2025. AMEX: Android Multi- annotation Expo Dataset for Mobile GUI Agents. InFindings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 2138–2156

    2025

  6. [6]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, et al. [n. d.]. GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding. InThe Thirteenth International Conference on Learning Representations

  7. [7]

    Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. 2025. PaddleOCR 3.0 Technical Report. arXiv:2507.05595 [cs.CV] https://arxiv.org/abs/2507.05595

    Pith/arXiv arXiv 2025

  8. [8]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 28091–28114

    2023

  9. [9]

    2025.DingTalk User Manual

    Dingtalk. 2025.DingTalk User Manual. https://alidocs.dingtalk.com/i/p/ dBgX4oYWY1aaEz8e

    2025

  10. [10]

    Liliana Dobrica. 2022. Robotic process automation platform UiPath.Commun. ACM65, 4 (March 2022), 42–43

    2022

  11. [11]

    Yifei Gao, Junhong Ye, Jiaqi Wang, and Jitao Sang. 2025. WebSynthe- sis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis. arXiv:2507.04370 [cs.AI]

    Pith/arXiv arXiv 2025

  12. [12]

    Peter Hofmann, Caroline Samp, and Nils Urbach. 2020. Robotic process automa- tion.Electronic Markets30, 1 (March 2020), 99–106

    2020

  13. [13]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14281–14290

    2024

  14. [14]

    Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingx- ing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. 2019. Searching for mobilenetv3. InProceedings of the IEEE/CVF international conference on computer vision. 1314–1324

    2019

  15. [15]

    Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. [n. d.]. OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use. ([n. d.])

  16. [16]

    Alejandro Jaimes and Nicu Sebe. 2007. Multimodal human–computer interaction: A survey.Computer vision and image understanding108, 1-2 (2007), 116–134

    2007

  17. [17]

    John and David E

    Bonnie E. John and David E. Kieras. 1996. Using GOMS for user interface design and evaluation: which technique?ACM Trans. Comput.-Hum. Interact.3, 4 (Dec. 1996), 287–319

    1996

  18. [18]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Vi- sualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 881–905

    2024

  19. [19]

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024. On the Effects of Data Scale on UI Control Agents. InAdvances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 92130–92154

    2024

  20. [20]

    Musen Lin, Minghao Liu, Taoran Lu, Lichen Yuan, Yiwei Liu, Haonan Xu, Yu Miao, Yuhao Chao, and Zhaojian Li. 2025. GUI-ReWalk: Massive Data Gener- ation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning. arXiv:2509.15738 [cs.LG] https://arxiv.org/abs/2509.15738

    arXiv 2025

  21. [21]

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang

  22. [22]

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. 2025. Guiodyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 22404– 22414

    2025

  23. [23]

    Xing Han Lu, Zdeněk Kasner, and Siva Reddy. 2024. WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. InInternational Conference on Machine Learning. PMLR, 33007–33056

    2024

  24. [24]

    nightly/daily builds

    A. Memon, I. Banerjee, N. Hashmi, and A. Nagarajan. 2003. DART: a framework for regression testing "nightly/daily builds" of GUI applications. InInternational Conference on Software Maintenance, 2003. ICSM 2003. Proceedings.410–419

    2003

  25. [25]

    Donald A. Norman. 2013.The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York

    2013

  26. [26]

    OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv. org/abs/2303.08774

    Pith/arXiv arXiv 2024

  27. [27]

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Ya...

    Pith/arXiv arXiv 2025

  28. [28]

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E Bishop, Wei Li, Folawiyo Campbell- Ajala, et al . [n. d.]. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. InThe Thirteenth International Conference on Learning Representations

  29. [29]

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang

  30. [30]

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. 2025. Os- genesis: Automating gui agent trajectory construction via reverse task synthesis. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 5555–5579

    2025

  31. [31]

    Karlsson, Bo An, and Zongqing Lu

    Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, and Zongqing Lu. 2024. Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study. InICLR 2024 Workshop on Large Language Model (LL...

    2024

  32. [32]

    Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. 2024. Omniparser: A unified framework for text spotting key information extraction and table recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15641–15653

    2024

  33. [33]

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, and Jianye Hao. 2025. GUI Agents with Foundation Models: A Comprehensive Survey. arXiv:2411.04890 [cs.AI] https://arxiv.org/ abs/2411.04890

    Pith/arXiv arXiv 2025

  34. [34]

    Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. 2025. Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks.arXiv preprint arXiv:2501.11733(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al . [n. d.]. OS- ATLAS: Foundation Action Model for Generalist GUI Agents. InThe Thirteenth International Conference on Learning Representations

  36. [36]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang

  37. [37]

    Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, et al. [n. d.]. UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems

  38. [38]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu

  39. [39]

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. 2025. Androidlab: Training and systematic benchmarking of android autonomous agents. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2144–2166

    2025

  40. [40]

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. InInternational Conference on Machine Learning. PMLR, 69772–69805

    2025

  41. [41]

    An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. 2023. GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation. arXiv:2311.07562 [cs.CV] https://arxiv.org/abs/ 2311.07562

    Pith/arXiv arXiv 2023

  42. [42]

    InAdvances in Neural Information Processing Systems, Vol

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. InAdvances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 52040–52094

  43. [43]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. Conference’17, July 2017, Washington, DC, USA Zhuohang Fan1, Beichen Zhang 1∗, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang 1,2∗ 1Harbin Institute of Technology, Weihai, China 2Harbin Institute...

    2022

  44. [44]

    Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li. 2026. Tongui: Internet-scale trajecto- ries from multimodal web tutorials for generalized gui agents. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 12367–12375

    2026

  45. [45]

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. Large Language Model-Brained GUI Agents: A Survey. arXiv:2411.18279 [cs.AI] https://arxiv.org/abs/2411.18279 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    Pith/arXiv arXiv 2025

  46. [46]

    Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions. arXiv:2306.02224 [cs.AI] https: //arxiv.org/abs/2306.02224

    Pith/arXiv arXiv 2023

  47. [2017]

    In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol

    World of Bits: An Open-Domain Platform for Web-Based Agents. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70). PMLR, 3135–3144

  48. [2018]

    InInternational Conference on Learning Representations

    Reinforcement Learning on Web Interfaces using Workflow-Guided Explo- ration. InInternational Conference on Learning Representations

  49. [2024]

    arXiv:2407.01489 [cs.SE] https://arxiv.org/abs/2407.01489

    Agentless: Demystifying LLM-based Software Engineering Agents. arXiv:2407.01489 [cs.SE] https://arxiv.org/abs/2407.01489

    Pith/arXiv arXiv