Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

This paper claims that a GUI agent's capability is driven by systematic data engineering and interactive infrastructure, and that the resulting UItron model beats state-of-the-art agents on GUI perception, grounding, planning, and Chinese m

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A serious systems paper with a genuinely new Chinese GUI dataset and strong external benchmark results, but the headline Chinese-superiority claim leans on a self-built evaluation that needs third-party verification. the 5 major comments →

arxiv 2508.21767 v1 pith:7UY2MPOZ submitted 2025-08-29 cs.CV

UItron: Foundational GUI Agent with Advanced Perception and Planning

Zhixiong Zeng , Jing Huang , Liming Zheng , Wenkang Han , Yufeng Zhong , Lei Chen , Longrong Yang , Yingjie Chu
show 2 more authors
Yuzhi He Lin Ma
This is my paper
classification cs.CV
keywords GUI agentvision-language modelreinforcement learninggroundingplanningChinese mobile appsdata engineeringGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UItron is an open-source GUI agent, built on Qwen2.5-VL, that the authors train to perceive, ground, and plan in mobile and PC interfaces. The paper's central claim is that two ingredients—systematically engineered training data and an interactive environment linking real phones and computers—are the main drivers of GUI-agent capability. Three training stages are used: supervised fine-tuning for perception, supervised fine-tuning for planning, and a curriculum reinforcement-learning stage with GRPO that moves from dense per-step rewards to task-level online rewards. The authors report leading results on standard benchmarks and a particularly large gain on their own Chinese-app evaluation, where UItron-72B reaches 47.4% task success versus 32.8% for UI-TARS-72B. The paper also argues that Chinese mobile apps are a blind spot for current state-of-the-art agents and that manually annotating one million action steps across the top 100 Chinese apps closes much of that gap.

Core claim

On its own terms, the paper establishes that a foundational GUI agent can be produced by combining (1) unified multi-task perception data, (2) planning data with layered reasoning, backtracking, and structured think/action output, (3) distilled trajectory data from an executable computer environment filtered by strict multi-model voting, and (4) a curriculum RL stage that first densifies rewards at the step level and then judges whole trajectories with multiple VLMs. Trained this way, UItron-72B outperforms the UI-TARS-72B baseline on ScreenSpot (90.3 vs 88.4), ScreenSpot-V2 (92.0 vs 90.3), AndroidControl (92.9 vs 88.5), and GUI-Odyssey, and reaches 24.9% success on OSWorld versus 22.7% for

What carries the argument

The load-bearing mechanism is the three-stage training pipeline. Stage 1 fine-tunes the vision-language backbone on perception tasks (grounding, captioning, VQA, OCR) with multi-turn consolidated and cross-platform unified data from open grounding datasets. Stage 2 fine-tunes on planning: next-action prediction with L1-L2-L3 reasoning levels, backtracking that forces the model to reconstruct how the current state arose, and an explicit <think>/<tool_call> output format. Stage 3 applies curriculum GRPO, first with dense per-step rewards on offline trajectories (easy), then task-level 0/1 rewards judged by multiple VLMs in an online environment (hard), with trajectories kept only if all scorer

Load-bearing premise

The headline Chinese result rests on the authors' own offline benchmark of 109 tasks (545 steps) and, for online success, on GPT-4o judging whether a task was finished; if that benchmark is unrepresentative or the judge rewards format rather than outcome, the claimed Chinese advantage may not transfer to real use.

What would settle it

Run UItron-72B and UI-TARS-72B on an independently created set of tasks in Chinese apps outside the top-100 list used for training, then have human raters score whether each task was actually completed, and check whether UItron's reported 14.6-point offline task-success lead (47.4 vs 32.8) survives.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training a GUI agent is primarily a data-engineering and infrastructure problem; the UItron recipe of perception and planning SFT followed by curriculum GRPO can be reused by others on top of an open VLM.
  • Large manually annotated trajectories from a target region or domain, here Chinese apps, can close a capability gap that general-purpose training leaves open.
  • Curriculum RL that moves from dense per-step rewards to trajectory-level VLM-judged rewards improves online exploration without hand-crafted reward functions.
  • The same open-source model achieves competitive online computer control on OSWorld (24.9%), narrowing the gap between GUI agents and specialized computer-use agents.
  • Because historical images can be omitted without significant degradation, GUI agents can be made cheaper at inference by relying on recorded historical actions alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed online-better-than-offline pattern (54.1% vs 47.4% for UItron-72B) suggests that offline step-success metrics understate real-world recoverability, since offline one failed step kills the episode; reporting both numbers together makes recovery ability visible.
  • The Chinese benchmark is private, so the 14.6-point lead is not independently checkable until the evaluation set or a matched public surrogate is released.
  • If the Chinese gain really comes from the manual million-step corpus, comparable corpus-building for other under-served languages and regions becomes the obvious scaling path for GUI agents—testable by ablating the Chinese data volume.
  • The paper's own future-work note that thinking and action outputs are frequently inconsistent indicates the reasoning format is only partially learned; directly rewarding thinking-action consistency could be the next training target.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces UItron, an open-source GUI agent built on Qwen2.5-VL and trained with a three-stage pipeline: GUI perception SFT, planning SFT, and a curriculum reinforcement-learning stage based on GRPO. The authors describe systematic data engineering (multi-turn/multi-task unification, cross-platform grounding data, trajectory distillation, VLM-voting filters, and over one million manually annotated Chinese-app action steps) and an interactive infrastructure for mobile and PC. They report strong results on GUI perception, grounding, offline planning, OSWorld, and newly constructed Chinese offline/online benchmarks, claiming superior performance and significant progress in Chinese app scenarios.

Significance. If the claims hold, the paper is a substantial systems contribution: UItron is open-sourced, includes detailed data-engineering methodology, a large manually annotated Chinese trajectory corpus, and a cloud-device interactive infrastructure. The 72B model is extensively compared against state-of-the-art GUI agents across perception, grounding, offline planning, and online planning. The main value lies in the practical recipe for building a generalist GUI agent and in the emphasis on non-English (Chinese) app coverage. However, the headline 'significant progress in Chinese app scenarios' currently rests on a small, self-built evaluation benchmark and a GPT-4o-based online judge, both of which raise robustness concerns that must be resolved before the broad claims are convincing.

major comments (5)
  1. [Section 4.5 / Table 9] The central Chinese-superiority claim is carried by a self-constructed benchmark of 109 manually annotated tasks (545 steps) with no inter-annotator agreement, no released test set, and only an assertion that test tasks do not overlap training tasks. Since UItron was trained on over one million steps from the same top-100 Chinese apps, the 47.4 vs 32.8 Task SR gap over UI-TARS-72B could reflect annotation-style overfitting or distributional leakage rather than task competence. The exact-match Task SR metric is especially sensitive to annotation conventions such as coordinate selection, action granularity, and step decomposition. Please release the test set and annotation protocol, report inter-annotator agreement, and provide a formal decontamination analysis.
  2. [Section 3.4.3 / Section 4.5 online evaluation] The online RL stage uses multiple VLM scorers to filter trajectories and compute rewards, and the online Chinese evaluation uses GPT-4o to judge task completion. This creates a potential circularity: the model is optimized to satisfy a GPT-4o-class judge, and the same class of judge is then used to certify success. Table 10 may therefore overstate real interactive competence. Please add an independent human evaluation of the online trajectories, report human–GPT-4o agreement, and use a judge that was not involved in training.
  3. [Section 3.2.3 / Section 4.4] The OSWorld evaluation is compromised by training/evaluation overlap. The trajectory-distillation stage explicitly uses the initial states of the 369 OSWorld tasks as prompts to generate training tasks, and executes trajectories in the OSWorld environment. Evaluating on OSWorld (Table 8, UItron-72B 24.9 vs UI-TARS-72B 22.7) is therefore not an out-of-distribution transfer test; the model has been exposed to the same environment distribution. Please either report OSWorld results using tasks/environments never touched during training or provide a detailed decontamination analysis showing that no training episodes share app states, initial screens, or task semantics with the evaluation episodes.
  4. [Section 3.1] The authors justify the important design decision to omit historical images by stating 'we empirically found that omitting historical images does not result in a significant performance degradation in most benchmarks,' but no supporting ablation is reported. This claim is load-bearing for the input format and efficiency. Please include a controlled comparison on at least AndroidControl-High and GUI-Odyssey showing performance with and without historical images, ideally at multiple history lengths.
  5. [Tables 5–8 and Tables 9–10] All reported numbers appear to be single runs without error bars, confidence intervals, or significance tests. Several headline margins are small (e.g., ScreenSpot-V2 92.0 vs 90.3, OSWorld 24.9 vs 22.7, AndroidControl-High grounding 94.2 vs 81.5). For the broad 'superior performance' claim, at least a few multiple-seed runs or bootstrap intervals should be provided so readers can assess whether the reported differences are meaningful.
minor comments (5)
  1. [Section 4.2] The text refers to 'Table 4.1' when discussing ScreenSpot-V2 results; this should be the corresponding numbered table (Table 6).
  2. [Section 4.2] The sentence 'UItron-72B relies solely on open-source data' is inconsistent with Sections 3.2.3 and 3.2.5, where the model uses trajectory distillation data and over one million manually annotated Chinese-app steps. Please clarify that this claim applies only to the grounding-training data used in a particular comparison.
  3. [Section 3.3] The interactive infrastructure is introduced with 'as shown in Figure 3', but Figure 3 appears to be the data-engineering figure; Figure 4 is the interactive-infrastructure figure. Cross-references should be corrected.
  4. [Section 2.2 / References] OS-Genesis is cited as [77], but [77] is Aria-UI. Please verify the citation and add the correct reference for OS-Genesis.
  5. [Throughout] There are repeated typos and inconsistencies: 'AndroidContorl' in Table 2, 'MutiUI' vs 'MultiUI', 'Augvis' vs 'Aguvis'/'AGUVIS', 'benchamrks' in Section 4.1, 'centra idea' in Section 3.4.2, and 'UItron conduct' in Section 5. A careful proofreading pass is needed.

Circularity Check

1 steps flagged

OSWorld is used both to generate training trajectories and as the evaluation benchmark, making one reported planning result partially circular; core perception/grounding claims rest on independent external benchmarks.

specific steps
  1. fitted input called prediction [Section 3.2.3 (Distillation Data) / Section 3.4.3 (Online RL) / Section 4.4 (OSWorld evaluation, Table 8)]
    "we utilized the initial states of 369 existing tasks in Osworld as prompts to generate additional, related yet distinct tasks using the GPT-4o extension ... The complete execution trajectory and task details are then evaluated by a VLM ... data that succeeded in a single attempt were utilized for SFT, while data requiring multiple attempts were identified as challenging cases and used for GRPO training. ... We use OSWorld [70] to evaluate the performance of GUI agent models as online agents ... It includes a benchmark of 369 tasks."

    OSWorld is used twice in the same paper: once as the source of training data and once as the evaluation benchmark. Section 3.2.3 generates tasks from the initial states of the 369 OSWorld tasks and distills trajectories in the OSWorld environment, with successful trajectories used for SFT and hard cases for GRPO; Section 3.4.3 further runs online RL in this interactive environment. Section 4.4 then evaluates on the same OSWorld benchmark of 369 tasks. Hence the Table 8 Task SR is not an independent measure: the model was trained on trajectories from the same environment and the same initial states that constitute the test set. The 'related yet distinct tasks' caveat does not break the shared initial-state/environment identity, so this reported planning result is partly forced by the traini

full rationale

The paper's headline Chinese-scenario claim is based on a self-constructed benchmark, but the paper asserts the 109 test tasks do not overlap training tasks and uses exact-match evaluation; that is a reproducibility/validity concern, not a demonstrated circular reduction. The Chinese online evaluation uses GPT-4o as judge while online RL uses unnamed multiple VLM scorers; because the paper does not state that GPT-4o is one of the RL scorers, I do not count this as a proven circular step, though it is a risk. The perception and grounding claims are supported by external benchmarks (VisualWebBench, ScreenSpot, RefExp, WidgetCap, WebSRC) and the offline planning claims by AndroidControl and GUI-Odyssey; these are independent and not circular. The one concrete circularity is OSWorld: the same 369-task benchmark is used to seed training-task generation and trajectory distillation, and then serves as the evaluation benchmark. This partially contaminates the online planning result in Table 8, but OSWorld is not the central claim and other benchmarks remain independent. A minor self-citation [27] for back-tracking exists but is not load-bearing. Overall partial circularity warrants 4/10.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central performance claim rests on training-data choices and evaluation design rather than on a mathematical derivation. No novel physical or mathematical entity is introduced. The effective free parameters are the unreported training hyperparameters and data mixtures.

free parameters (3)
  • GRPO hyperparameters (epsilon, beta, group size G) = not reported
    The GRPO objective (Eq. 3) uses a clip range epsilon, KL coefficient beta, and a group of G sampled responses; no values are given, yet the RL gain depends on them.
  • Training data mixture ratios = not reported
    The paper describes mixing perception, planning, distillation, general multimodal, and Chinese data qualitatively (Sections 3.2.1-3.2.5) but never states the proportions, which are load-bearing for the reported benchmark gains.
  • History prompt level (L2) = L2 reasoning descriptions
    Section 3.2.2 states L2-level descriptions are used as historical context during inference; this is a design choice that affects planning accuracy and is not ablated.
axioms (4)
  • domain assumption Qwen2.5-VL is a suitable base model whose capabilities are retained after GUI fine-tuning
    All UItron versions are built on Qwen2.5-VL weights (Section 3.4 Summary); if the base model were the main driver rather than the training, the attribution of gains would be clouded.
  • ad hoc to paper The claim that omitting historical images does not significantly degrade performance
    Section 3.1 states this as an empirical finding but provides no experiment; the whole architecture relies on this to reduce computation.
  • domain assumption Open-source GUI datasets (Uground, Aria-UI, Aguvis, OS-Atlas) can be unified into one format without loss
    Section 3.2.1 says the datasets are integrated with different synthesis criteria; any format mismatch could affect grounding quality.
  • domain assumption VLM voting provides a correct measure of trajectory success
    Sections 3.2.3 and 3.4.3 filter trajectories by agreement of multiple VLM scorers; if these scorers are systematically wrong, both SFT and RL data are contaminated.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of UItron: Foundational GUI Agent with Advanced Perception and Planning." pith.science (2026). https://pith.science/paper/7UY2MPOZ

@misc{pith2026250821767,
  author       = {Pith},
  title        = {Pith review of: UItron: Foundational GUI Agent with Advanced Perception and Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UY2MPOZ}},
  note         = {Machine review of arXiv:2508.21767}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

GUI agent aims to enable automated operations on Mobile/PC devices, which is an important task toward achieving artificial general intelligence. The rapid advancement of VLMs accelerates the development of GUI agents, owing to their powerful capabilities in visual understanding and task planning. However, building a GUI agent remains a challenging task due to the scarcity of operation trajectories, the availability of interactive infrastructure, and the limitation of initial capabilities in foundation models. In this work, we introduce UItron, an open-source foundational model for automatic GUI agents, featuring advanced GUI perception, grounding, and planning capabilities. UItron highlights the necessity of systemic data engineering and interactive infrastructure as foundational components for advancing GUI agent development. It not only systematically studies a series of data engineering strategies to enhance training effects, but also establishes an interactive environment connecting both Mobile and PC devices. In training, UItron adopts supervised finetuning over perception and planning tasks in various GUI scenarios, and then develop a curriculum reinforcement learning framework to enable complex reasoning and exploration for online environments. As a result, UItron achieves superior performance in benchmarks of GUI perception, grounding, and planning. In particular, UItron highlights the interaction proficiency with top-tier Chinese mobile APPs, as we identified a general lack of Chinese capabilities even in state-of-the-art solutions. To this end, we manually collect over one million steps of operation trajectories across the top 100 most popular apps, and build the offline and online agent evaluation environments. Experimental results demonstrate that UItron achieves significant progress in Chinese app scenarios, propelling GUI agents one step closer to real-world application.

Figures

Figures reproduced from arXiv: 2508.21767 by Jing Huang, Lei Chen, Liming Zheng, Lin Ma, Longrong Yang, Wenkang Han, Yingjie Chu, Yufeng Zhong, Yuzhi He, Zhixiong Zeng.

Figure 1
Figure 1. Figure 1: Comparison of UItron and UI-Tars in perception/grounding/planning and Chinese scenarios. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The core capabilities of GUI agent, including GUI perception, grounding, offline planning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overall introduction of data engineering. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall introduction of interactive infrastructure. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The overall architecture of Mobile infra. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overall introduction of training paradigm. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

    cs.AI 2026-06 unverdicted novelty 6.0

    GUICrafter uses curriculum learning on unannotated GUI screenshots for visual grounding followed by RL calibration on limited labels to match or exceed prior GUI agents with far less annotation.

  2. GUI Agents with Reinforcement Learning: Toward Digital Inhabitants

    cs.AI 2026-04 unverdicted novelty 5.0

    The paper delivers the first comprehensive overview of RL for GUI agents, organizing methods into offline, online, and hybrid strategies while analyzing trends in rewards, efficiency, and deliberation to outline a fut...

Reference graph

Works this paper leans on

87 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716–23736, 2022

    2022

  3. [3]

    Introducing claude 4.5

    Anthropic. Introducing claude 4.5. URL https://www.anthropic.com/news/claude-4, 2025

    2025

  4. [4]

    Uibert: Learning generic multimodal representations for ui understanding

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107.13731, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Less is more: Empowering gui agent with context-aware simplification

    Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kaiwen Zhou, Shuai Wang, Wentao Li, Yinchuan Li, Zhongang Qi, and Liqiang Nie. Less is more: Empowering gui agent with context-aware simplification. In International Conference on Computer Vision, 2025

    2025

  8. [8]

    Allava: Harnessing gpt4v- synthesized data for lite vision-language models

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684, 2024. 18

    Pith/arXiv arXiv 2024

  9. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  10. [10]

    Websrc: A dataset for web-based structural reading comprehension

    Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websrc: A dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences , 67(12):220101, 2024

    2024

  12. [12]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    2024

  13. [13]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems , 36:49250–49267, 2023

    2023

  15. [15]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  16. [16]

    Agentic reinforced policy optimization

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Anytool: Self-reflective, hierarchical agents for large-scale api calls

    Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls. In International Conference on Machine Learning , pages 11812–11829. PMLR, 2024

    work page 2024

  18. [18]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Clova: A closed-loop visual assistant with tool usage and update

    Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13258–13268, 2024

    work page 2024

  20. [20]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations , 2025

    work page 2025

  22. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025. 19

    Pith/arXiv arXiv 2025

  24. [24]

    A real-world webagent with planning, long context understanding, and program synthesis

    Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. In ICLR, 2024

    work page 2024

  25. [25]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 6864–6890, 2024

    work page 2024

  26. [26]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14281–14290, 2024

    work page 2024

  27. [27]

    Scaletrack: Scaling and back-tracking automated gui agents

    Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, and Lin Ma. Scaletrack: Scaling and back-tracking automated gui agents. arXiv preprint arXiv:2505.00416, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Autowebglm: A large language model- based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model- based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining , pages 5295–5306, 2024

    work page 2024

  30. [30]

    On the effects of data scale on ui control agents

    Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    2024

  31. [31]

    Widget captioning: Generating natural language description for mobile user interface elements

    Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprint arXiv:2010.04295, 2020

    Pith/arXiv arXiv 2010

  32. [32]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26763–26773, 2024

    work page 2024

  33. [33]

    Showui: One vision-language-action model for gui visual agent

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weix- ian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 19498–19508, 2025

    work page 2025

  34. [34]

    Robust function-calling for on-device language model via function masking

    Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, et al. Robust function-calling for on-device language model via function masking. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  35. [35]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024

    2024

  37. [37]

    Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    2024

  38. [38]

    Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023. 20

    work page 2023

  39. [39]

    Harnessing webpage uis for text-rich visual understanding

    Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding. arXiv preprint arXiv:2410.13824, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners

    Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504.14239, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. In International Conference on Computer Vision, 2025

    work page 2025

  44. [44]

    Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning

    Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Cheap and quick: Efficient vision-language instruction tuning for large language models

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems, 36:29615–29627, 2023

    work page 2023

  46. [46]

    Gui-r1: A generalist r1-style vision-language action model for gui agents

    Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Operator

    OpenAI. Operator. URL https://openai.com/research/operator, 2025

    work page 2025

  48. [48]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744, 2022

    2022

  49. [49]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems , 36:53728–53741, 2023

    work page 2023

  51. [51]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Gui-xplore: Empowering generalizable gui agents with one exploration

    Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. Gui-xplore: Empowering generalizable gui agents with one exploration. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 19477–19486, 2025

    work page 2025

  54. [54]

    Gui-g2: Gaussian reward modeling for gui grounding

    Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. Gui-g2: Gaussian reward modeling for gui grounding. arXiv preprint arXiv:2507.15846, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Kimi k2: Open agentic intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. 21

    Pith/arXiv arXiv 2025

  56. [56]

    Appworld: A controllable world of apps and people for benchmarking interactive coding agents

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers),...

    work page 2024

  57. [57]

    Llms in the imagi- narium: Tool learning through simulated trial and error

    Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. Llms in the imagi- narium: Tool learning through simulated trial and error. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , pages 10583–10604, 2024

    work page 2024

  58. [58]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023

    Pith/arXiv arXiv 2023

  59. [59]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems , 37:121475–121499, 2024

    2024

  61. [61]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems , 36:61501–61513, 2023

    work page 2023

  62. [62]

    Opencua: Open foundations for computer-use agents

    Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, et al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.09123, 2025

    arXiv 2025

  63. [63]

    Self-instruct: Aligning language models with self-generated instruc- tions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022

    Pith/arXiv arXiv 2022

  64. [64]

    Trove: inducing verifiable and efficient toolboxes for solving programmatic tasks

    Zora Zhiruo Wang, Graham Neubig, and Daniel Fried. Trove: inducing verifiable and efficient toolboxes for solving programmatic tasks. In Proceedings of the 41st International Conference on Machine Learning, pages 51177–51191, 2024

    work page 2024

  65. [65]

    Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning

    Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning. arXiv preprint arXiv:2505.16421, 2025

    arXiv 2025

  66. [66]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Os-atlas: Foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: Foundation action model for generalist gui agents. In The Thirteenth International Conference on Learning Representations , 2025

    work page 2025

  68. [68]

    Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent

    Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2025. 22

    work page 2025

  70. [70]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    2024

  71. [71]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    Pith/arXiv arXiv 2024

  72. [72]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. In International Conference on Machine Learning , 2025

    work page 2025

  73. [73]

    Zerogui: Automating online gui learning at zero human cost

    Chenyu Yang, Shiqian Su, Shi Liu, Xuan Dong, Yue Yu, Weijie Su, Xuehui Wang, Zhaoyang Liu, Jinguo Zhu, Hao Li, et al. Zerogui: Automating online gui learning at zero human cost. arXiv preprint arXiv:2505.23762, 2025

    Pith/arXiv arXiv 2025

  74. [74]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Pith/arXiv arXiv 2023

  75. [75]

    Gpt4tools: Teaching large language model to use tools via self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36:71995–72007, 2023

    2023

  76. [76]

    Aria-ui: Visual grounding for gui instructions

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024

    Pith/arXiv arXiv 2024

  77. [77]

    Aria- UI: Visual grounding for GUI instructions

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria- UI: Visual grounding for GUI instructions. In Findings of the Association for Computational Linguistics: ACL 2025 , pages 22418–22433, 2025

    work page 2025

  78. [78]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    Pith/arXiv arXiv 2023

  79. [79]

    Easytool: Enhancing llm-based agents with concise tool instruction

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Kan Ren, Dongsheng Li, and Deqing Yang. Easytool: Enhancing llm-based agents with concise tool instruction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pages...

    work page 2025

  80. [80]

    Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning

    Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. arXiv preprint arXiv:2505.12370, 2025

    Pith/arXiv arXiv 2025

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.