Pith. sign in

REVIEW 4 major objections 4 minor 4 cited by

ZeroGUI: Automating Online GUI Learning at Zero Human Cost

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ZeroGUI claims that a fully automated online loop — VLM-generated tasks, VLM-estimated rewards, and two-stage reinforcement learning — improves GUI agents on OSWorld and AndroidLab with no human annotations.

desk verdict Worth reading as an engineering contribution, but the headline generalization claim rests on training on the test tasks themselves, which the paper openly reports but does not confront. read the letter →

arxiv 2505.23762 v1 pith:Q6QA5ZMB submitted 2025-05-29 cs.AI cs.CLcs.CV

Chenyu Yang , Shiqian Su , Shi Liu , Xuan Dong , Yue Yu , Weijie Su , Xuehui Wang , Zhaoyang Liu
show 6 more authors
Jinguo Zhu Hao Li Wenhai Wang Yu Qiao Xizhou Zhu Jifeng Dai
This is my paper
classification cs.AIcs.CLcs.CV
keywords GUIagentsonlinereinforcementlearningvision-languagemodelsautomatictaskgenerationrewardestimationtest-timetrainingOSWorldAndroidLab
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ZeroGUI claims that GUI agents can be trained entirely online, with no human annotation and no hand-written success verifiers. The framework lets a vision-language model propose diverse tasks from screenshots, lets the agent attempt them, and uses another vision-language model to score each attempt from the full screenshot trajectory. On OSWorld, applying this loop to UI-TARS-7B-DPO raises success from 17.7% to 20.2%, and to Aguvis-7B from 3.0% to 4.9%; on AndroidLab, it raises UI-TARS-7B-DPO from 45.7% to 47.5% on the full test set. If correct, this removes the main bottleneck to scaling GUI agents: expensive human-labeled trajectories.

What carries the argument

The machinery is a closed loop: a VLM proposes tasks from a random initial screenshot, the GUI agent rolls out trajectories, and a second VLM, queried four times on all screenshots with the agent's textual responses removed and unanimous voting for success, assigns binary rewards. The policy is updated with Group Relative Policy Optimization (GRPO) extended to multi-step trajectories, with the KL term replaced by a per-token MSE estimator, $\frac{1}{2}(\log\pi_\theta - \log\pi_{\mathrm{ref}})^2$, which the paper argues stabilizes training. The two stages — generated-task training and test-time training — play complementary roles: the first expands capability coverage, the second improves consistency on target tasks.

What would settle it

Take the exact test tasks used in the second training stage, run the trained agents to collect trajectories, have humans label success and failure on those trajectories, and compare with the VLM's unanimous-vote reward. If agreement is low, or if false positives concentrate on the tasks where success rates rose, then the training signal rather than the policy would be the likely source of the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fully automated online training framework, ZeroGUI, can improve GUI agents without any human labels, hand-crafted tasks, or hand-written verifiers. The framework generates training tasks by prompting a VLM with screenshots and exemplars, estimates binary success rewards by querying another VLM on all trajectory screenshots with unanimous voting, and updates the policy through a two-stage online reinforcement learning procedure: first on generated tasks, then on the test tasks themselves at test time. The authors report that this raises UI-TARS-7B-DPO success on OSWorld from 17.7% to 20.2% (14% relative, and 40% relative on the feasible subset) and Aguvis-7B from 3.0% to 4.9% (63% relative), with complementary gains in pass@4 and all-pass@4 metrics, and that the method transfers to AndroidLab with a +2.8 success-rate gain on the operation subset.

Load-bearing premise

The whole improvement rests on a vision-language model being able to judge, from screenshots alone, whether a generated or test task actually succeeded — accurately enough that reinforcement learning is guided by true success rather than by the model's mistakes.

Editorial extensions

If this is right

  • GUI-agent training can proceed in environments that provide only screenshots and an action interface, without collecting or labeling offline trajectories.
  • Test-time training can adapt an agent to a benchmark's actual test tasks even when no ground-truth verifier exists, because the VLM serves as the verifier.
  • Because both UI-TARS-7B-DPO and Aguvis-7B improved, the loop generalizes across different base models and across desktop and mobile environments.
  • The reward estimator's precision-over-recall design implies that false-positive rewards are the main training risk, so reward estimators for this setting should prioritize precision.
  • Generated-task training and test-time training are complementary: the former mainly improves pass@4 capability coverage, the latter mainly improves all-pass@4 consistency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the VLM reward is accurate enough, the same loop could be pointed at any interactive environment that exposes screenshots and an action API, so GUI-agent training could scale to new applications without new benchmark verifiers.
  • The paper validates reward precision and recall only on generated-task trajectories; a natural extension would be to measure reward agreement on the exact test trajectories used in stage two and to monitor whether agents learn actions that fool the VLM without completing the task.
  • The pass@4 and all-pass@4 results suggest that the two training stages could be tuned separately, for example by varying the ratio of generated to test-time tasks to control the tradeoff between capability breadth and behavioral consistency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ZeroGUI, a fully automated online learning framework for GUI agents. It uses a VLM to generate training tasks from environment screenshots, a VLM-based reward estimator to provide binary success signals without manual verifiers, and a two-stage GRPO-style RL procedure: first on generated tasks, then on the test tasks themselves at test time. Experiments on OSWorld and AndroidLab with UI-TARS-7B-DPO and Aguvis-7B report success-rate improvements, e.g., UI-TARS from 17.7 to 20.2 on OSWorld full test set and from 45.7 to 47.5 on AndroidLab. The appendix includes a derivation of a k2-KL variant of GRPO that stabilizes training.

Significance. If the results withstand scrutiny, the framework is a valuable step toward annotation-free GUI-agent training: it combines automatic task generation, automatic reward estimation, and online RL in a single loop, and the k2-KL modification is clearly derived and appears technically sound. The code is released, which supports reproducibility. However, the significance is currently tempered by the evaluation protocol (test-time training on the exact benchmark tasks) and by the limited validation of the reward estimator on the test distribution; these issues affect the strength of the generalization claims rather than the validity of the core idea.

major comments (4)
  1. [Sec. 4.2.1, Table 1] The reported end-to-end results include a second RL stage performed on the exact OSWorld test tasks, so the headline improvement mixes test-set adaptation with general capability gains. For UI-TARS-7B-DPO on the full test set, the generated-task-only stage reaches SR 18.2 (base 17.7±1.1), the test-time-only stage reaches 18.2, and the full two-stage system reaches 20.2; on the feasible subset the corresponding numbers are 14.7, 14.4, and 15.8. Thus the generated-task stage alone is within one standard deviation of the base model on the full set, and the final gain is largely produced by the interaction of both stages. The claim in Sec. 3.3 that generated-task training 'builds general capabilities' would be substantially strengthened by evaluating the two-stage pipeline on a held-out task split that is disjoint from the test-time training tasks, or by clearly separating the benchmark-adaptation result from a transfer result.
  2. [Sec. 4.3, Table 3b] The reward estimator is validated only on trajectories from generated tasks, with precision and recall measured on a manually labeled set. No such validation is provided for the exact test-set trajectories used in the second training stage, even though the second stage relies entirely on VLM-estimated rewards for these tasks. A systematic bias in the reward estimator on the test distribution (for example, false positives that reward incomplete executions) could inflate the reported success rates. The authors should report reward-estimator precision/recall on a human-labeled sample of test-task trajectories, and where possible compare VLM rewards against the environment verifiers on the subset of tasks with deterministic verifiers.
  3. [Sec. 4.3, Tables 3-4, Fig. 3] The ablation results are reported as single numbers without variance or significance information, while the main tables report mean±std over 4 runs. The differences underlying several claims are small: k2-KL vs. k3-KL is 27.2 vs. 26.1, online RL vs. online RFT is 27.2 vs. 24.5, and the task-generation ablations span 22.3 to 27.2. Without run-to-run variance it is difficult to judge whether these gaps are real. Please report multiple seeds for the ablation tables and figure, or at least provide the number of runs used.
  4. [Sec. 3.1, Appendix C.1] The task-generation prompt uses 'instruction exemplars' (Sec. 3.1) but the paper never states where these exemplars come from. If they are drawn from the OSWorld or AndroidLab test sets, the generated-task distribution is conditioned on the evaluation distribution, which would weaken the generalization claim and could also bias the VLM reward estimator toward the test domain. The source of the exemplars should be disclosed, and the discussion in Sec. 4.2.1 and Appendix C.1 should be updated accordingly.
minor comments (4)
  1. [Sec. 4.1.1, Sec. 1] The word 'caculated' should be 'calculated', and in Sec. 1 'pure-vison-based' should be 'pure-vision-based'.
  2. [Sec. 4.1.2, Fig. 6] The KL coefficient β=0.1 is selected via the sweep in Fig. 6; it would be helpful to state explicitly that this sweep is performed on the Daily domain of OSWorld, consistent with the other ablations.
  3. [Table 2 note] The sentence 'we fix some code errors in the original task verifiers' needs more detail: which verifiers were changed and how do the corrections affect comparability with previously published AndroidLab numbers?
  4. [Appendix B.1, Eq. (8)] Equation (8) simplifies the objective under the assumption πθ = πθold, which holds in the one-update-per-rollout setting; stating this assumption explicitly before the simplification would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: training rewards and evaluation verifiers are distinct; the test-time-training design is disclosed and weakens generalization claims but is not definitionally circular.

full rationale

I walked the derivation chain: tasks are generated by GPT-4o from exemplars and an initial screenshot; rewards come from Qwen2.5-VL unanimous voting; policy updates use GRPO with a k2-KL term; final scores come from the OSWorld and AndroidLab scripted verifiers (and GPT scoring for the AndroidLab query subset). The trained reward estimator is used only to supply RL supervision, while the reported success rates are computed by the benchmarks' external verifiers, so the training objective is not identical to the evaluation metric by construction. No parameter is fitted to the target SR, and no load-bearing self-citation or imported uniqueness theorem is used to force the method. The one notable concern is that the second stage performs test-time training on the exact test-task instructions and then evaluates on those same tasks, which limits the claim of general transfer and should be read as transductive adaptation rather than held-out generalization. This is an experimental-design and interpretation issue, not a circular-reasoning defect: the evaluation still uses independent environment checks, and the paper does not rename this adaptation as an unprompted prediction. Under the stated criteria, the derivation is self-contained and the paper's central improvements, while partially attributable to test-task adaptation, do not reduce to the method's own inputs by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of the VLM-based reward estimator and the quality of VLM-generated tasks, both of which are domain assumptions rather than derived results. The free parameters include the KL coefficient, the training-task sample size, and the reward voting rule, all tuned by the authors. No new physical or conceptual entities are introduced.

free parameters (3)
  • KL coefficient beta = 0.1
    Ablated in Fig 6; 0.1 gives best test SR; removing KL or using 0.01 or 1.0 degrades performance.
  • Number of training tasks sampled = 725 Ubuntu, 175 Android
    Randomly sampled from generated pool, approximately twice the size of test sets; set by the authors.
  • VLM reward voting rule = 4 queries, unanimous agreement
    Chosen to reduce false positives (Tab 3b); a stricter rule than majority voting.
assumptions (5)
  • domain assumption VLM task generation (GPT-4o) produces feasible, diverse tasks that match the operational constraints of the target environments.
    Sec 3.1: tasks are generated from screenshots and exemplars; if generated tasks are infeasible or too narrow, training may not transfer.
  • domain assumption VLM reward estimates correlate with true task success on the evaluation benchmarks.
    Sec 3.2: the reward VLM is used for all training rewards; precision/recall is measured only on a small set of generated-task trajectories (Tab 3b), not on test tasks.
  • standard math The MDP formulation of GUI tasks with state = (observation, history) and action sequences is valid.
    Eq. 1-2; standard RL framing.
  • ad hoc to paper Test-time training on the exact test-set tasks (with VLM rewards) is an acceptable evaluation protocol.
    Sec 3.3: the second stage uses test-set tasks; this is disclosed but unusual for benchmark evaluation, as it trains on the evaluation tasks.
  • standard math The k2-KL estimator provides a stable approximation of the KL divergence for GRPO.
    Appendix B provides derivation; they show k2 avoids large gradients compared to k3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZeroGUI: Automating Online GUI Learning at Zero Human Cost." pith.science (2026). https://pith.science/paper/Q6QA5ZMB

@misc{pith2026250523762,
  author       = {Pith},
  title        = {Pith review of: ZeroGUI: Automating Online GUI Learning at Zero Human Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6QA5ZMB}},
  note         = {Machine review of arXiv:2505.23762}
}
read the original abstract

The rapid advancement of large Vision-Language Models (VLMs) has propelled the development of pure-vision-based GUI Agents, capable of perceiving and operating Graphical User Interfaces (GUI) to autonomously fulfill user instructions. However, existing approaches usually adopt an offline learning framework, which faces two core limitations: (1) heavy reliance on high-quality manual annotations for element grounding and action supervision, and (2) limited adaptability to dynamic and interactive environments. To address these limitations, we propose ZeroGUI, a scalable, online learning framework for automating GUI Agent training at Zero human cost. Specifically, ZeroGUI integrates (i) VLM-based automatic task generation to produce diverse training goals from the current environment state, (ii) VLM-based automatic reward estimation to assess task success without hand-crafted evaluation functions, and (iii) two-stage online reinforcement learning to continuously interact with and learn from GUI environments. Experiments on two advanced GUI Agents (UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance across OSWorld and AndroidLab environments. The code is available at https://github.com/OpenGVLab/ZeroGUI.

Figures

Figures reproduced from arXiv: 2505.23762 by the authors.

Figure 1
Figure 1. Left: Existing Offline Training Framework for GUI Agents incurs high human costs, relying on manually collected and annotated interaction trajectories, typically under a supervised fine-tuning (SFT) paradigm. Right: Our ZeroGUI is a scalable online learning framework with automated task generation and reward estimation at zero human cost. A VLM proposes diverse tasks, which are executed by the agent; the agent then … view at source ↗
Figure 2
Figure 2. Top: Overview of ZeroGUI. It adopts a Two-stage Online Reinforcement Learning paradigm. In the first stage, tasks are automatically generated by a VLM, while in the second stage, tasks are drawn from the test set. These tasks are executed by the GUI agent. After each interaction, a reward is assigned automatically by the VLM based on the agent’s trajectory, and the policy network is updated via reinforcement learnin… view at source ↗
Figure 3
Figure 3. , k2-KL yields higher and more stable training accuracy. Test success rates in Tab. 4 further validate the superiority of k2-KL in our setting [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Gradient coefficient of KL loss. 0 5 10 15 20 25 30 Training Steps 0.00 0.05 0.10 0.15 0.20 0.25 KL loss GRPO (k3-KL) Ours (k2-KL) (a) KL loss. 0 5 10 15 20 25 30 Training Steps 0 5 10 15 20 25 30 m a x(lo g lo g ref) GRPO (k3-KL) Ours (k2-KL) (b) max(log πθ − log πref…
Figure 5
Figure 5. Figure 5: KL loss curve and token-wise maximum and minimum of [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Test success rates with different KL loss coefficients [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the test and generated task instructions in OSWorld. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The trajectory of the base model UI-TARS-7B-DPO when executing a VS Code task. The instruction is “I want the tabs to wrap onto multiple lines when they exceed the available space. Please help modify the VS Code settings accordingly.” The original thoughts are in Chine…
Figure 9
Figure 9. Figure 9: The trajectory of our trained model when executing a VS Code task. The instruction is “I want the tabs to wrap onto multiple lines when they exceed the available space. Please help modify the VS Code settings accordingly.” The original thoughts are in Chinese and have …
Figure 10
Figure 10. Figure 10: The trajectory of the base model UI-TARS-7B-DPO when executing a LibreOffice Impress task. The instruction is “Add an image ‘none.png’ on the Desktop to slide 2 with 1cm*1cm size.” The original thoughts are in Chinese and have been translated into English for presenta…
Figure 11
Figure 11. Figure 11: The trajectory of our trained model when executing a LibreOffice Impress task. The instruction is “Add an image ‘none.png’ on the Desktop to slide 2 with 1cm*1cm size.” The original thoughts are in Chinese and have been translated into English for presentation purpose…
Figure 12
Figure 12. Figure 12: The trajectory of the base model UI-TARS-7B-DPO when executing a Calendar task. The instruction is “You should use calendar to complete the following task: Arrange an event titled ‘homework’ for me at May 21st, and set the notification time to be 10 minutes before.” 2…
Figure 13
Figure 13. Figure 13: The trajectory of our trained model when executing a Calendar task. The instruction is “You should use calendar to complete the following task: Arrange an event titled ‘homework’ for me at May 21st, and set the notification time to be 10 minutes before.” 23 [PITH_FUL…

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlowBA: An efficiency backdoor attack towards VLM-based GUI agents

    cs.CR 2026-03 conditional novelty 7.0 of 10

    SlowBA uses two-stage reward-level injection and pop-up triggers to make VLM GUI agents produce much longer, slower responses under attack while largely preserving task accuracy.

  2. SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

    cs.AI 2026-07 conditional novelty 6.5 of 10

    A compact multi-agent judge with a shared 9B backbone matches or beats rule-based reward signals in online RL for computer-use agents, per the authors' held-out success-rate measurements.

  3. UItron: Foundational GUI Agent with Advanced Perception and Planning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

  4. MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment

    cs.LG 2025-07 conditional novelty 5.0 of 10

    MobileGUI-RL applies online RL with self-generated and filtered tasks plus trajectory-level rewards to mobile GUI agents, reporting improved success rates on AndroidWorld and AITW benchmarks.

Reference graph

Works this paper leans on

84 extracted references · 17 canonical work pages · cited by 4 Pith papers

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Introducing claude 3.5 sonnet

    Anthropic. Introducing claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet ,

  3. [3]

    H. Bai, Y . Zhou, J. Pan, M. Cemri, A. Suhr, S. Levine, and A. Kumar. Digirl: Training in-the-wild device- control agents with autonomous reinforcement learning.Advances in Neural Information Processing Systems, 37:12461–12495, 2024

    2024

  4. [4]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  5. [5]

    R. Bellman. A markovian decision process.Journal of mathematics and mechanics, pages 679–684, 1957

    1957

  6. [6]

    Bonatti, D

    R. Bonatti, D. Zhao, F. Bonacci, D. Dupont, S. Abdali, Y . Li, Y . Lu, J. Wagle, K. Koishida, A. Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264, 2024

    arXiv 2024

  7. [7]

    Y . Chai, S. Huang, Y . Niu, H. Xiao, L. Liu, D. Zhang, P. Gao, S. Ren, and H. Li. Amex: Android multi-annotation expo dataset for mobile gui agents.arXiv preprint arXiv:2407.17490, 2024

    arXiv 2024

  8. [8]

    L. Chen, L. Li, H. Zhao, Y . Song, and Vinci. R1-v: Reinforcing super generalization ability in vision- language models with less than $3. https://github.com/Deep-Agent/R1-V , 2025. Accessed: 2025- 02-02

    2025

Show all 84 references
  1. [9]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  2. [10]

    W. Chen, J. Cui, J. Hu, Y . Qin, J. Fang, Y . Zhao, C. Wang, J. Liu, G. Chen, Y . Huo, et al. Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024

    2024 arXiv

  3. [11]

    Cheng, Q

    K. Cheng, Q. Sun, Y . Chu, F. Xu, Y . Li, J. Zhang, and Z. Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  4. [12]

    H. Deng, D. Zou, R. Ma, H. Luo, Y . Cao, and Y . Kang. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning.arXiv preprint arXiv:2503.07065, 2025

    2025 arXiv

  5. [13]

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  6. [14]

    Drouin, M

    A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    2024 arXiv

  7. [15]

    D. Gao, L. Ji, Z. Bai, M. Ouyang, P. Li, D. Mao, Q. Wu, W. Zhang, P. Wang, X. Guo, et al. Assistgui: Task-oriented pc graphical user interface automation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13289–13298, 2024

    2024

  8. [16]

    Gemini 1.5 pro | generative ai on vertex ai

    Google. Gemini 1.5 pro | generative ai on vertex ai. https://cloud.google.com/vertex-ai/ generative-ai/docs/models/gemini/1-5-pro, 2024. Accessed: 2025-05-11

    2024

  9. [17]

    B. Gou, R. Wang, B. Zheng, Y . Xie, C. Chang, Y . Shu, H. Sun, and Y . Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  10. [18]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  11. [19]

    H. He, W. Yao, K. Ma, W. Yu, Y . Dai, H. Zhang, Z. Lan, and D. Yu. Webvoyager: Building an end-to-end web agent with large multimodal models.arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  12. [20]

    W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y . Wang, Z. Wang, Y . Dong, M. Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024. 24

    2024

  13. [21]

    Hsiao, F

    Y .-C. Hsiao, F. Zubach, G. Baechler, V . Carbune, J. Lin, M. Wang, S. Sunkara, Y . Zhu, and J. Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots.arXiv preprint arXiv:2209.08199, 2022

    2022 arXiv

  14. [22]

    Huang, B

    W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y . Hu, and S. Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  15. [23]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [24]

    Kapoor, Y

    R. Kapoor, Y . P. Butala, M. Russak, J. Y . Koh, K. Kamble, W. AlShikh, and R. Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pages 161–178. Springer, 2024

    2024

  17. [25]

    J. Y . Koh, R. Lo, L. Jang, V . Duvvur, M. C. Lim, P.-Y . Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  18. [26]

    H. Li, J. Chen, J. Su, Y . Chen, Q. Li, and Z. Zhang. Autogui: Scaling gui grounding with automatic functionality annotations from llms.arXiv preprint arXiv:2502.01977, 2025

    2025 arXiv

  19. [27]

    W. Li, W. E. Bishop, A. Li, C. Rawles, F. Campbell-Ajala, D. Tyamagundlu, and O. Riva. On the effects of data scale on ui control agents.Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    2024

  20. [28]

    Y . Li, J. He, X. Zhou, Y . Zhang, and J. Baldridge. Mapping natural language instructions to mobile ui action sequences.arXiv preprint arXiv:2005.03776, 2020

    2005 arXiv

  21. [29]

    K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, W. Lei, L. Wang, and M. Z. Shou. Showui: One vision-language-action model for gui visual agent.arXiv preprint arXiv:2411.17465, 2024

    2024 arXiv

  22. [30]

    E. Z. Liu, K. Guu, P. Pasupat, T. Shi, and P. Liang. Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    2018 arXiv

  23. [31]

    J. Liu, Y . Song, B. Y . Lin, W. Lam, G. Neubig, Y . Li, and X. Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955, 2024

    2024 arXiv

  24. [32]

    X. Liu, B. Qin, D. Liang, G. Dong, H. Lai, H. Zhang, H. Zhao, I. L. Iong, J. Sun, J. Wang, et al. Autoglm: Autonomous foundation agents for guis.arXiv preprint arXiv:2411.00820, 2024

    2024 arXiv

  25. [33]

    X. Liu, T. Zhang, Y . Gu, I. L. Iong, Y . Xu, X. Song, S. Zhang, H. Lai, X. Liu, H. Zhao, et al. Visualagent- bench: Towards large multimodal models as visual foundation agents.arXiv preprint arXiv:2408.06327, 2024

    2024 arXiv

  26. [34]

    Y . Liu. Rethinking kl divergence in rlhf: From single sample to mini-batch to expecta- tion. https://www.notion.so/Rethinking-KL-Divergence-in-RLHF-From-Single-Sample-to-Mini-Batch-to- Expectation-1c18637cdeb3800ab47cd01d3fa33ea5, 2025. Notion Blog

    2025

  27. [35]

    Y . Liu, P. Li, C. Xie, X. Hu, X. Han, S. Zhang, H. Yang, and F. Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners.arXiv preprint arXiv:2504.14239, 2025

    2025 arXiv

  28. [36]

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  29. [37]

    Z. Liu, Z. Sun, Y . Zang, X. Dong, Y . Cao, H. Duan, D. Lin, and J. Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  30. [38]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  31. [39]

    Q. Lu, W. Shao, Z. Liu, F. Meng, B. Li, B. Chen, S. Huang, K. Zhang, Y . Qiao, and P. Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  32. [40]

    X. H. Lù, Z. Kasner, and S. Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930, 2024

    2024

  33. [41]

    Z. Lu, Y . Chai, Y . Guo, X. Yin, L. Liu, H. Wang, G. Xiong, and H. Li. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620, 2025. 25

    2025 arXiv

  34. [42]

    McInnes, J

    L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  35. [43]

    Introducing operator

    OpenAI. Introducing operator. https://openai.com/index/introducing-operator/, 2025. Ac- cessed: 2025-05-16

    2025

  36. [44]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    2022

  37. [45]

    Y . Pan, D. Kong, S. Zhou, C. Cui, Y . Leng, B. Jiang, H. Liu, Y . Shang, S. Zhou, T. Wu, et al. Webcanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373, 2024

    2024 arXiv

  38. [46]

    Y . Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  39. [47]

    Z. Qi, X. Liu, I. L. Iong, H. Lai, X. Sun, W. Zhao, Y . Yang, X. Yang, J. Sun, S. Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning.arXiv preprint arXiv:2411.02337, 2024

    2024 arXiv

  40. [48]

    Y . Qin, Y . Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y . Li, S. Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  41. [49]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36: 53728–53741, 2023

    2023

  42. [50]

    Rawles, A

    C. Rawles, A. Li, D. Rodriguez, O. Riva, and T. Lillicrap. Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    2023

  43. [51]

    Rawles, S

    C. Rawles, S. Clinckemaillie, Y . Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell- Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  44. [52]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  45. [53]

    Schulman

    J. Schulman. Approximating kl divergence. http://joschu.net/blog/kl-approx.html, 2020. Ac- cessed: 2025-05-16

    2020

  46. [54]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  47. [55]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  48. [56]

    H. Shen, P. Liu, J. Li, C. Fang, Y . Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  49. [57]

    T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. InInternational Conference on Machine Learning, pages 3135–3144. PMLR, 2017

    2017

  50. [58]

    L. Sun, X. Chen, L. Chen, T. Dai, Z. Zhu, and K. Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui.arXiv preprint arXiv:2205.11029, 2022

    2022 arXiv

  51. [59]

    Q. Sun, K. Cheng, Z. Ding, C. Jin, Y . Wang, F. Xu, Z. Wu, C. Jia, L. Chen, Z. Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.arXiv preprint arXiv:2412.19723, 2024

    2024 arXiv

  52. [60]

    K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  53. [61]

    J. Wan, S. Song, W. Yu, Y . Liu, W. Cheng, F. Huang, X. Bai, C. Yao, and Z. Yang. Omniparser: A unified framework for text spotting key information extraction and table recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1564...

    2024

  54. [62]

    J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception.arXiv preprint arXiv:2401.16158, 2024. 26

    2024 arXiv

  55. [63]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y . Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  56. [64]

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y . Wang, J. Ji, Z. Yang, L. Zhao, S. XiXuan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Systems, 37: 121475–121499, 2024

    2024

  57. [65]

    H. Wen, Y . Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y . Liu, Y . Zhang, and Y . Liu. Autodroid: Llm-powered task automation in android. InProceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543–557, 2024

    2024

  58. [66]

    Z. Wu, Z. Wu, F. Xu, Y . Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al. Os-atlas: A foundation action model for generalist gui agents.arXiv preprint arXiv:2410.23218, 2024

    2024 arXiv

  59. [67]

    Xia and R

    X. Xia and R. Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458, 2025

    2025 arXiv

  60. [68]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    2024

  61. [69]

    T. Xu, L. Chen, D.-J. Wu, Y . Chen, Z. Zhang, X. Yao, Z. Xie, Y . Chen, S. Liu, B. Qian, et al. Crab: Cross- environment agent benchmark for multimodal language model agents.arXiv preprint arXiv:2407.01511, 2024

    2024 arXiv

  62. [70]

    Y . Xu, X. Liu, X. Sun, S. Cheng, H. Yu, H. Lai, S. Zhang, D. Zhang, J. Tang, and Y . Dong. Androidlab: Training and systematic benchmarking of android autonomous agents.arXiv preprint arXiv:2410.24024, 2024

    2024 arXiv

  63. [71]

    Y . Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  64. [72]

    Y . Yang, Y . Wang, D. Li, Z. Luo, B. Chen, C. Huang, and J. Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  65. [73]

    Y . Yang, X. He, H. Pan, X. Jiang, Y . Deng, X. Yang, H. Lu, D. Yin, F. Rao, M. Zhu, et al. R1- onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  66. [74]

    S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  67. [75]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  68. [76]

    Q. Yu, Z. Zhang, R. Zhu, Y . Yuan, X. Zuo, Y . Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  69. [77]

    W. Yu, Z. Yang, J. Wan, S. Song, J. Tang, W. Cheng, Y . Liu, and X. Bai. Omniparser v2: Structured-points- of-thought for unified visual text parsing and its generality to multimodal large language models.arXiv preprint arXiv:2502.16161, 2025

    2025 arXiv

  70. [78]

    Zhang, J

    J. Zhang, J. Wu, Y . Teng, M. Liao, N. Xu, X. Xiao, Z. Wei, and D. Tang. Android in the zoo: Chain-of- action-thought for gui agents.arXiv preprint arXiv:2403.02713, 2024

    2024 arXiv

  71. [79]

    H. H. Zhao, D. Gao, and M. Z. Shou. Worldgui: Dynamic testing for comprehensive desktop gui automation. arXiv preprint arXiv:2502.08047, 2025

    2025 arXiv

  72. [80]

    Zheng, B

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  73. [81]

    aha moment

    H. Zhou, X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model.arXiv preprint arXiv:2503.05132, 2025

    2025 arXiv

  74. [82]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  75. [83]

    Y . Zuo, K. Zhang, S. Qu, L. Sheng, X. Zhu, B. Qi, Y . Sun, G. Cui, N. Ding, and B. Zhou. Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084, 2025. 27

    2025 arXiv

  76. [2024]

    Accessed: 2025-05-11

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.