Pith. sign in

REVIEW 4 major objections 5 minor 44 references

TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark measures how GUI agents transfer across versions, platforms, and apps.

desk verdict A genuinely new benchmark idea whose headline cross-platform result is undercut by an app-identity confound; worth refereeing, but the authors need to restructure the experiments. read the letter →

arxiv 2505.17629 v2 pith:TAQ7CRRD submitted 2025-05-23 cs.HC cs.AI

Yuheng Lu , Qian Yu , Hongru Wang , Zeming Liu , Wei Su , Yanping Liu , Yuhang Guo , Maocheng Liang
show 2 more authors
Yunhong Wang Haifeng Wang
This is my paper
classification cs.HCcs.AI
keywords GUIagentsgroundingtransferabilitybenchmarkcross-versioncross-platformcross-applicationfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that GUI agents fail in practice not because they cannot ground instructions on a single screen, but because they cannot transfer that grounding across app versions, platforms, and applications. To make this measurable, it introduces TransBench, a benchmark covering 15 app categories, 81 apps, and 1,459 screenshots across Android old and new versions, iOS, and the Web, with more than 22,000 human-verified grounding instructions. The central experiment fine-tunes Aria-UI on low-version Android data and shows accuracy gains on new Android, iOS, and Web, suggesting that historical UI data can prepare agents for future interface changes. If true, this turns version and platform drift from a threat into a training signal.

What carries the argument

The load-bearing object is TransBench itself: a dataset built in three stages — screenshot acquisition from Android old and new versions, iOS, and Web for 81 common apps; automated bounding-box annotation with OmniParser followed by manual verification of semantic equivalence; and instruction generation where Qwen2VL produces per-element visual, positional, and functional attributes plus screen summaries, and Qwen-plus turns those into user-style instructions, all filtered through human quality control that reports 95.5% agreement. The dataset's metadata — app name, page title, app version, platform — is what lets the authors partition it into training and test splits that isolate each transferability dimension.

What would settle it

Re-run the core fine-tuning experiment with a subset of TransBench instructions that are independently rewritten by human users on the same screenshots; if the accuracy gains from low-version Android fine-tuning disappear or shrink sharply on that subset, the reported transferability effect is an artifact of LLM-style instruction wording rather than genuine grounding transfer.

Watch

Extended reading notes

Core claim

TransBench is the first benchmark to simultaneously evaluate three transferability dimensions of GUI grounding: cross-version, cross-platform, and cross-application. Using it, the paper shows that fine-tuning Aria-UI on 5,000 low-version Android screenshots raises grounding accuracy on new-version Android from 81.38% to 88.36%, on iOS from 77.61% to 82.57%, and on the Web from 66.86% to 73.61% — gains that come from data that contains no new-version, iOS, or Web samples at all. The authors interpret this as evidence that grounding knowledge generalizes along version and platform axes, and that benchmark construction with explicit version, platform, and app metadata is what makes such transfer measurable.

Load-bearing premise

The benchmark's ground truth is only as reliable as its automated generation pipeline: if the LLM-produced instructions systematically name elements in ways that only match its own UI understanding, every accuracy number measured on TransBench inherits that bias despite the 95.5% human agreement score.

Editorial extensions

If this is right

  • Fine-tuning a GUI grounding model on older-version data is a viable strategy for adapting to app updates without waiting for new-version annotations.
  • Android data may serve as a strong generic training source: it transfers to iOS and Web better than iOS data transfers to other platforms.
  • Cross-application transfer is real but weaker than cross-platform and cross-version transfer; app-category similarity matters less than version and platform alignment.
  • Web grounding remains the hardest transfer target, and meaningful Web gains appear to require mixing multi-platform data during training.
  • The released benchmark gives a standard protocol for comparing transferability of future GUI agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to regenerate TransBench-style instructions from real human-typed queries on the same screenshots; if the fine-tuning gains shrink, the reported transferability may be partly an artifact of LLM-style instruction wording rather than pure grounding transfer.
  • The results suggest a continual-learning loop: as apps update, automatically mine old-version screenshots (which are abundant and cheap) to keep agents current, reducing dependence on freshly labeled data.
  • The three transferability axes could be combined into a single transfer matrix that predicts how much annotation effort on one platform saves on others; TransBench's metadata already supports computing such a matrix.
  • The paper's scaled Euclidean distance metric could be adopted as a standard complement to accuracy in GUI grounding evaluations, since accuracy alone misses near-miss predictions that still indicate understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. TransBench introduces a GUI-grounding benchmark covering 1,459 screenshots from 81 apps across Android (old and new versions), iOS, and Web, with over 65,000 bounding boxes and about 22,000 human-checked instructions. The paper defines three transferability dimensions for grounding: cross-version, cross-platform, and cross-application, and evaluates several grounding models (CogAgent, SeeClick, Aria-UI, OS-Atlas, UGround, Qwen2.5VL) on accuracy and a proposed normalized-distance metric. It then fine-tunes Aria-UI on different partitions of the benchmark to study transferability, reporting gains such as Android-new accuracy increasing from 81.38% to 88.36% after fine-tuning on low-version Android data, with smaller gains on iOS and Web.

Significance. If the claims hold, TransBench is a useful resource: it is one of the first GUI-grounding benchmarks to include version and platform metadata and to attempt a systematic study of transferability, and it includes a detailed data-collection pipeline, human verification, and a complementary distance metric. The paper also evaluates a diverse set of current grounding models, which gives the community a useful comparison point. The main strength is the dataset design; the main weakness is that the headline cross-platform transferability claim is not cleanly isolated from app identity. Because the underlying data and code are not yet released, independent verification of the benchmark and the fine-tuning results is currently impossible.

major comments (4)
  1. [Section 5.1 / Appendix B.2] The cross-platform transferability evidence is confounded by app identity. The Android-Low training set (Appendix B.2) is a random 5,000-sample subset of 5,696 low-version Android samples, and these samples come from the same apps that also contribute the iOS and Web test sets (Table 2 shows 77 Android apps, 81 iOS apps, and 47 Web apps with substantial overlap). Consequently, the gains on iOS and Web in Table 5 could reflect app-specific memorization (e.g., learning Taobao's layout and icons from Android screenshots and applying them to iOS/Web Taobao screenshots) rather than platform-agnostic transfer. To support the claim of cross-platform transferability, the authors should report an experiment in which the test apps are disjoint from the training apps, or at least condition the analysis on app overlap and show that gains persist for apps never seen during training.
  2. [Section 5.3 / Appendix B.2] The cross-application experiment does not control for platform. The App Partition in Appendix B.2 selects apps from the top seven categories and holds out 40% of the apps, but no platform restriction is stated. If the training and test samples are drawn from multiple platforms, the improvements in Figure 3 could be driven by platform or version adaptation rather than by application-level transfer. The authors should either restrict the training and test sets to a single platform or report per-platform breakdowns for the same-category and different-category results.
  3. [Section 5.1 / Table 5 / Figure 3] The fine-tuning conclusions rest on a single model, a single seed, and no confidence intervals or statistical tests. The paper uses the word 'significant' (e.g., Section 5.1 and Figure 3) to describe accuracy improvements of 4.96 and 6.69 percentage points, but without repeated runs, variance estimates, or significance tests, these gains cannot be distinguished from noise. At minimum, the authors should report multiple fine-tuning runs with the same and different seeds, or provide error bars, before claiming significant transferability improvements.
  4. [Section 3.4] The ground-truth quality-control report is underspecified. The paper states that human evaluation yields an 'average score of 95.5%', but it does not report the sample size, the number of annotators per item, inter-annotator agreement, or how disagreements were resolved. Because the instructions are generated by Qwen-plus from Qwen2VL-derived summaries and attributes, a systematic bias in the automated stages could affect every grounding accuracy number measured on TransBench. The authors should describe the verification protocol in detail and, ideally, release the data so that independent checks are possible.
minor comments (5)
  1. [Appendix A.1] The sentence introducing the prompt details uses 'Tabel' instead of 'Table'; please fix the typo.
  2. [Table 1 and References] VisualAgentBench appears twice in the reference list (Liu et al., 2024b and Liu et al., 2024c) with identical titles; please merge or remove the duplicate.
  3. [Section 3.3 / Figure 2] The pipeline figure is dense and the font is small; consider enlarging the text and making the three steps more visually distinct.
  4. [Section 4.2 / Equation (1)] The definition of distance D uses a norm symbol with a comma inside the double bars, which is nonstandard; please write the Euclidean distance explicitly.
  5. [Appendix B.3] The paper says 'we trained Lora' and later 'Aria-ui', but the model name and the acronym LoRA should be spelled consistently throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: TransBench reports empirical benchmark measurements, and its transferability gains are not constructed to equal their inputs.

full rationale

TransBench is a measurement and benchmark-construction paper rather than a derivation chain. Its central outputs—grounding accuracy on Android, iOS, and Web, and accuracy changes from fine-tuning Aria-UI—are direct empirical measurements on held-out splits, not quantities derived from fitted parameters or from equations that contain their own conclusions. The fine-tuning results in Tables 5–7 and Figure 3 compare a base model before and after training; nothing in the construction forces the post-fine-tuning accuracy to equal the training labels or to be a renamed version of any input statistic. The new distance metric in Eq. (1) is an evaluation definition, not a circular derivation. The paper does cite prior work, including some by its own authors (e.g., Liu et al. 2020 and Wang et al. 2024d in quality-control references), but those citations are not load-bearing for the transferability claims. The strongest concern—that Android-Low training samples and iOS/Web test samples come from overlapping applications—is a potential experimental confound about whether the measured gains reflect app-specific memorization rather than general transfer, but it is not circularity: the reported gains are not definitionally forced by the benchmark design. The 'first benchmark' claim is a novelty assertion, not a theorem imported by self-citation. Since no prediction reduces by construction to an input, fitted value, or self-citation chain, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The benchmark's core value depends on the accuracy and unbiasedness of the annotation pipeline. All downstream conclusions about transferability inherit the quality of these assumptions.

free parameters (2)
  • LoRA hyperparameters (r, alpha, dropout, lr, batch size, epochs) = 8, 32, 0.05, 5e-5, 16, 2
    Chosen by hand for all fine-tuning experiments in Section 5; Figure 8 checks epoch count on one split, but no search or sensitivity analysis is reported.
  • Training set size per partition = 5,000 (Web: 4,000)
    Each transferability experiment randomly selects 5,000 training samples (Section B.2); the choice of this size is arbitrary and could affect the magnitude of the reported gains.
assumptions (5)
  • domain assumption OmniParser autonomously detected GUI elements with sufficient accuracy to serve as initial bounding-box ground truth.
    Used in Step 2 (Section 3.3) before manual verification; if OmniParser misses or mislabels elements, the verified boxes inherit those errors.
  • domain assumption Qwen2VL-generated attributes and summaries are semantically faithful to the screenshots.
    Used in Section 3.4 to produce visual/positional/functional descriptions that later ground instruction generation.
  • domain assumption Qwen-plus-generated instructions match user intent uniquely to the target element.
    Section 3.4 uses Qwen-plus to write instructions; the 95.5% human check is asserted but no inter-annotator reliability is reported.
  • domain assumption The 81 apps and page selection are representative of real-world GUI use.
    Section 3.2 chooses apps from 'previous studies and everyday usage'; the claim of representativeness is not empirically justified.
  • domain assumption Grounding accuracy (click within bounding box) is a valid measure of GUI agent grounding.
    Section 4.2 defines the metric; it inherits the standard assumption used by prior benchmarks that box containment corresponds to successful grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments." pith.science (2026). https://pith.science/paper/TAQ7CRRD

@misc{pith2026250517629,
  author       = {Pith},
  title        = {Pith review of: TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TAQ7CRRD}},
  note         = {Machine review of arXiv:2505.17629}
}
read the original abstract

Graphical User Interface (GUI) agents, which autonomously operate on digital interfaces through natural language instructions, hold transformative potential for accessibility, automation, and user experience. A critical aspect of their functionality is grounding - the ability to map linguistic intents to visual and structural interface elements. However, existing GUI agents often struggle to adapt to the dynamic and interconnected nature of real-world digital environments, where tasks frequently span multiple platforms and applications while also being impacted by version updates. To address this, we introduce TransBench, the first benchmark designed to systematically evaluate and enhance the transferability of GUI agents across three key dimensions: cross-version transferability (adapting to version updates), cross-platform transferability (generalizing across platforms like iOS, Android, and Web), and cross-application transferability (handling tasks spanning functionally distinct apps). TransBench includes 15 app categories with diverse functionalities, capturing essential pages across versions and platforms to enable robust evaluation. Our experiments demonstrate significant improvements in grounding accuracy, showcasing the practical utility of GUI agents in dynamic, real-world environments. Our code and data will be publicly available at GitHub.

Figures

Figures reproduced from arXiv: 2505.17629 by the authors.

Figure 1
Figure 1. Interpretation of Transferability’s three aspects. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Interpretation of data collection process. The blue box represents our proposed benchmark - [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Sub-figure (a), (b) shows the variation of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An example of Android version. Manage the current shopping cart interface by clicking the three dots at the top right of the screen. Search for items in your shopping cart by entering their name in the search box at the top of the shopping cart page. Click the magnifyi…
Figure 5
Figure 5. Figure 5: An example of iOS version. Click 'mobile shopping Taobao', enter the mobile shopping Taobao page. Click on this orange "My Taobao“. Click to view the 3 items in the shopping cart. Click the favorites button on the right side of the navigation bar at the top of the shop…
Figure 6
Figure 6. Figure 6: An example of the Web version. A.3 Application Names and Page Titles [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Enter Caption A.4 Other details Tools details. To quickly inspect the generated data, we develop a manual inspection tool in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: Three examples of error cases [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 5 canonical work pages

  1. [1]

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. https://doi.org/10.24963/ijcai.2024/339 Screenai: A vision-language model for ui and infographics understanding . In Proceedings of the Thirty-Third International Joint Conference on Artificial Int...

    doi:10.24963/ijcai.2024/339 2024

  2. [2]

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Ag \"u era y Arcas. 2021. https://api.semanticscholar.org/CorpusID:236493482 Uibert: Learning generic multimodal representations for ui understanding . In International Joint Conference on Artificial Intelligence

    work page 2021

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. https://api.semanticscholar.org/CorpusID:261101015 Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    2023

  4. [4]

    Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022. A dataset for interactive vision language navigation with unknown command feasibility. In European Conference on Computer Vision (ECCV)

    work page 2022

  5. [5]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. 2024 a . https://api.semanticscholar.org/CorpusID:270560712 Gui-world: A dataset for gui-oriented multimodal llm-based...

    arXiv 2024

  6. [6]

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. 2024 b . Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317

    arXiv 2024

  7. [7]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

    arXiv 2024

  8. [8]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. 2023. Mobilevlm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886

    arXiv 2023

Show all 44 references
  1. [9]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    2024

  2. [10]

    Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. 2024 a . Assistgui: Task-oriented pc graphical user interface automation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  3. [11]

    Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. 2024 b . Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.14337

    2024

  4. [12]

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243

    2024 arXiv

  5. [13]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14...

    2024

  6. [14]

    Yongxiang Hu, Xuan Wang, Yingchuan Wang, Yu Zhang, Shiyu Guo, Chaoyi Chen, Xin Wang, and Yangfan Zhou. 2024. https://api.semanticscholar.org/CorpusID:271162246 Auitestagent: Automatic requirements oriented gui function testing . ArXiv, abs/2407.09018

    2024 arXiv

  7. [15]

    Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, et al. 2024. Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios. arXiv preprint arXiv:2401.17167

    2024 arXiv

  8. [16]

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2025. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pag...

    2025

  9. [17]

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. 2024. https://api.semanticscholar.org/CorpusID:273229053 Aria: An open multimodal native mixture-of-experts model . ArXiv, abs/2410.05993

    2024 arXiv

  10. [18]

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776

    2020 arXiv

  11. [19]

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. 2024 a . Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820

    2024 arXiv

  12. [20]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

    2023 arXiv

  13. [22]

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. 2024 c . Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327

    2024 arXiv

  14. [23]

    Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. https://doi.org/10.18653/v1/2020.acl-main.98 Towards conversational recommendation over multi-type dialogs . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis...

    2020 doi

  15. [24]

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024 a . Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451

    2024 arXiv

  16. [25]

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024 b . https://api.semanticscholar.org/CorpusID:271601072 Omniparser for pure vision based gui agent . ArXiv, abs/2408.00203

    2024 arXiv

  17. [26]

    Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.539 C o C o-agent: A comprehensive cognitive MLLM agent for smartphone GUI automation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 9097--9110, Bangk...

    2024 doi

  18. [27]

    Hamid Mukhtar. 2025. Artificial intelligence techniques for human-machine interaction. In Artificial Intelligence and Multimodal Signal Processing in Human-Machine Interaction, pages 19--42. Elsevier

    2025

  19. [28]

    Xiaoming Shi, Zeming Liu, Chuan Wang, Haitao Leng, Kui Xue, Xiaofan Zhang, and Shaoting Zhang. 2023. https://doi.org/10.18653/v1/2023.acl-long.453 M id M ed: Towards mixed-type dialogues for medical consultation . In Proceedings of the 61st Annual Meeting of the Association fo...

    2023 doi

  20. [29]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwen2.5-vl/ Qwen2.5-vl

    2025

  21. [30]

    Pan, and Kam-Fai Wong

    Hongru Wang, Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. 2024 a . https://doi.org/10.1145/3626772.3661381 Empowering large language models: Tool learning for real-world interaction . In Proceedings of the 47th International ACM SIGIR Conference on Research and Develo...

    2024

  22. [31]

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024 b . Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01014

    2024 arXiv

  23. [32]

    Ke Wang, Tianyu Xia, Zhangxuan Gu, Yi Zhao, Shuheng Shen, Changhua Meng, Weiqiang Wang, and Ke Xu. 2024 c . E-ant: A large-scale dataset for efficient automatic gui navigation. arXiv preprint arXiv:2406.14250

    2024 arXiv

  24. [33]

    Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. 2024 d . http://arxiv.org/abs/2410.12896 A survey on data synthesis and augmentation for large language models . ArXiv, abs/2410.12896

    2024 arXiv

  25. [34]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Ke-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 e . https://api.semanticscholar.org/Cor...

    2024 arXiv

  26. [35]

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, and Ruiming Tang. 2024 f . Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890

    2024 arXiv

  27. [36]

    Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. 2024 a . Mobilevlm: A vision-language model for better intra-and inter-ui understanding. arXiv preprint arXiv:2409.14818

    2024 arXiv

  28. [37]

    Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Liujian Liujianfeng, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.599 M obile VLM : A vision-language model for better intra- and inter- UI understanding . In Findings of the Asso...

    2024 doi

  29. [38]

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. 2024 c . Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218

    2024 arXiv

  30. [39]

    Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224

    2023 arXiv

  31. [40]

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. 2024. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256

    2024 arXiv

  32. [41]

    Team Glm Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Ming yue Liu, Minlie H...

    2024 arXiv

  33. [42]

    Rajmohan, Dongmei Zhang, and Qi Zhang

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Ming-Jie Ma, Qingwei Lin, S. Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. https://api.semanticscholar.org/CorpusID:274306375 Large language model-brained gui agents: A survey . ArXiv, abs/2411.18279

    2024 arXiv

  34. [43]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117--50143

    2023

  35. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.