Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception

T0 review · 3 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that a single end-to-end VLM, VLM-3D, can perform open-world 3D perception in autonomous driving, reporting a 12.8% accuracy gain on nuScenes — but the supplied full text is a different paper and contains no suc

desk verdict Title/abstract promise VLM-3D, but the full text is an unrelated MVISU-Bench manuscript; the submission has no content to referee. read the letter →

arxiv 2508.09061 v1 pith:ZNYBLVHK submitted 2025-08-12 cs.CV

Fuhao Chang , Shuxin Li , Yabei Li , Lei He This is my paper
classification cs.CV
keywords VLM-3Dopen-world3Dperceptionvision-languagemodelsautonomousdrivingLow-RankAdaptation(LoRA)jointsemantic-geometriclossopen-setdetectionnuScenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The abstract of this record proposes VLM-3D, described as the first end-to-end framework that lets a vision-language model (VLM) perform 3D geometric perception in autonomous driving. The intended contribution is to replace multi-stage pipelines — a VLM extracting features, a separate detector producing boxes — with a single model, using Low-Rank Adaptation (LoRA) for cheap fine-tuning and a two-phase joint semantic-geometric loss: token-level semantic loss for stable early convergence, then 3D IoU loss to sharpen bounding-box predictions. On nuScenes the abstract reports a 12.8% improvement in perception accuracy over the baseline without this loss. If that held, open-set detection of unseen object categories — a known safety bottleneck in driving — could be handled by one model rather than a cascade. The text supplied with this record, however, is a different manuscript (the MVISU-Bench mobile-agent benchmarking paper) and contains none of the VLM-3D architecture, training details, or nuScenes experiments.

What carries the argument

Two mechanisms carry the claimed result, both described only in the abstract. First, Low-Rank Adaptation (LoRA): a parameter-efficient fine-tuning method that trains small low-rank weight updates to adapt a pre-trained VLM to driving data at minimal overhead. Second, the joint semantic-geometric loss: a training schedule that applies token-level semantic loss early — anchoring category recognition and language alignment — and switches to 3D IoU loss later to sharpen predicted 3D bounding boxes against ground truth. The ordering is the point: semantic learning is meant to stabilize convergence before geometric regression is refined, and this schedule is what the abstract credits with the 12.8

What would settle it

Look in the supplied full text for the nuScenes evaluation table, the LoRA configuration, and a description (let alone a diagram) of the VLM-3D architecture; none of these appear — the text reports a 404-task benchmark of mobile agents across 137 apps. A version of the paper that still omits the nuScenes experiments, or whose tables do not reproduce the claimed 12.8% perception-accuracy gain under the joint semantic-geometric loss, would settle the claim.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a single end-to-end VLM can do open-world 3D perception in driving scenes: the model ingests camera input and outputs both semantic understanding and 3D bounding boxes, adapted by LoRA at minimal cost and trained with a joint semantic-geometric loss — token-level semantic loss early for stable convergence, 3D IoU loss later to refine box accuracy. The reported result is a 12.8% perception-accuracy gain on nuScenes, offered as evidence that an end-to-end design beats multi-stage coupling. The supplied full text does not corroborate this: it is the MVISU-Bench paper on benchmarking mobile agents, whose only overlap is incidental vocabulary (VLMs, LoR

Load-bearing premise

The load-bearing premise is that the supplied full text is the VLM-3D paper — that it contains the architecture, LoRA configuration, loss schedule, and nuScenes experiments behind the abstract's claims; the full text instead is the MVISU-Bench mobile-agent benchmarking paper, so if that premise gives way the central claim has no supporting material in this record.

Editorial extensions

If this is right

  • If the claim holds, open-world 3D detection in driving becomes a single forward pass of one vision-language model, with no separate detector module to compound errors.
  • Categories never seen during training could in principle be localized through language-grounded semantics rather than a closed set of class labels — the safety-critical case the paper motivates.
  • The two-phase loss schedule (token-level semantic loss first, 3D IoU loss later) would become a reusable template for turning VLMs into geometric regressors on other perception tasks.
  • The claimed 12.8% gain is the paper's evidence that end-to-end training beats multi-stage coupling; if replicated, it would motivate retiring feature-extractor-plus-detector stacks in driving perception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My reading: the abstract's framing presupposes that a VLM's language tokens and a 3D box head can be trained jointly without the language objective overwhelming geometric regression; the interesting open question is how the two loss phases should be scheduled or weighted per task.
  • A testable extension: apply the same recipe (LoRA plus semantic-then-IoU loss) to nuScenes with an open vocabulary and report gains separately for seen versus novel categories — a single 12.8% number cannot tell whether the benefit is open-world generalization or better box refinement on known classes.
  • My reading: the practical consequence of the manuscript mismatch is that anyone wanting to check the VLM-3D claims must obtain a complete version of the actual paper; nothing in the present record allows the architecture, loss, or nuScenes results to be inspected.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The submission carries the title 'VLM-3D: End-to-End Vision-Language Models for Open-World 3D Perception' and an abstract asserting a new end-to-end VLM framework for 3D detection in autonomous driving, with LoRA adaptation, a joint token-level semantic and 3D-IoU loss, and a 12.8% perception-accuracy improvement on nuScenes. The full text of the submitted document, however, is a different paper: 'MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions', an ACM MM '25 submission whose running header is 'arXiv:2508.09057v2 [cs.CL] 14 Aug 2025'. The body describes a mobile-agent benchmark, the Aider plug-in module, and experiments on mobile GUI agent success rates. It contains no VLM-3D architecture, no 3D geometric perception method, no nuScenes experiments, and no 3D IoU loss. The claimed contribution therefore has no supporting derivation, implementation, or evaluation in the submitted manuscript.

Significance. Had the manuscript actually contained the claimed VLM-3D framework, the contribution could be significant: an end-to-end VLM for 3D perception with a joint semantic-geometric loss would address a real limitation of multi-stage autonomous-driving perception pipelines. However, the submission as provided does not contain that work. The full text is a different manuscript about mobile GUI agents; while the MVISU-Bench benchmark and Aider module may be useful in their own area, they are unrelated to the stated central claim. The only verifiable claims in the submitted document are about mobile-agent success rates, not 3D perception. The paper cannot be evaluated as a 3D perception paper because none of its technical content is present.

major comments (3)
  1. [Title/Abstract vs. Full Text] The central claim—an end-to-end VLM-3D architecture with LoRA and a joint semantic/3D-IoU loss yielding a 12.8% improvement on nuScenes—is stated only in the title and abstract. The full text is a different manuscript, MVISU-Bench, as evidenced by the running header 'arXiv:2508.09057v2 [cs.CL] 14 Aug 2025' and the ACM MM '25 formatting. No section, equation, or table in the body describes VLM-3D, its loss, its LoRA adaptation, or its 3D bounding-box output. This is not a gap in presentation; it is an internal inconsistency between the asserted contribution and the artifact under review.
  2. [§5 (Experiments)] The experimental section reports success rates of mobile agent frameworks (Mobile-Agent, Mobile-Agent-V2, Mobile-Agent-E) on the MVISU-Bench tasks. There are no nuScenes experiments, no perception-accuracy metric, and no 3D IoU evaluation. The abstract's quantitative claim of a '12.8% improvement in perception accuracy' is therefore unsupported by any data in the manuscript; no table or figure corresponds to this number.
  3. [Appendix C.2] The only occurrence of LoRA in the body is for fine-tuning the Aider module on Qwen2.5-VL-3B for mobile-GUI assistance. This is unrelated to 3D perception and cannot be read as evidence for the VLM-3D framework. The manuscript thus provides no technical support for the abstract's architectural claims, and the correctness of the claimed method cannot be assessed.
minor comments (2)
  1. [Throughout] Because the body belongs to a different paper, the reference list, mathematical notation, and terminology are entirely mismatched with the abstract. Even considered as its own paper, the MVISU-Bench text contains typos such as 'the the process' (§3.1) and 'shonw' (Appendix C.2), which should be corrected if that manuscript is resubmitted separately.
  2. [Not applicable] No figure or table in the submitted document supports any claim in the abstract; this is a structural problem rather than a copy-editing issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation can be audited because the submitted full text is a different paper (MVISU-Bench); the VLM-3D claims are unsupported but not circular.

full rationale

The abstract promises VLM-3D, an end-to-end framework with LoRA adaptation and a joint semantic-geometric loss yielding a 12.8% improvement on nuScenes. However, the supplied full text is the MVISU-Bench paper, as evidenced by the page header 'arXiv:2508.09057v2 [cs.CL] 14 Aug 2025' and the ACM MM '25 formatting; it contains no VLM-3D architecture, no 3D detection formulation, no nuScenes experiments, and no derivation of the claimed improvement. There is therefore no derivation chain to walk: the central numerical claim is unsupported by the provided artifact, but unsupported evidence is not circular reasoning. No equation equates an output to an input, no fitted parameter is renamed as a prediction, and no load-bearing self-citation chain appears. The only internal inconsistency is between the asserted VLM-3D contribution and the actual MVISU-Bench body, which is a document-identity problem rather than a circularity pattern. Accordingly, the appropriate finding is no significant circularity (score 0), while noting that the VLM-3D claims are entirely unverified in this submission.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The manuscript body contains no equations, parameters, or new entities related to VLM-3D. The included MVISU-Bench text is a separate paper and cannot be used to audit the stated central claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception." pith.science (2026). https://pith.science/paper/ZNYBLVHK

@misc{pith2026250809061,
  author       = {Pith},
  title        = {Pith review of: VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNYBLVHK}},
  note         = {Machine review of arXiv:2508.09061}
}
read the original abstract

Open-set perception in complex traffic environments poses a critical challenge for autonomous driving systems, particularly in identifying previously unseen object categories, which is vital for ensuring safety. Visual Language Models (VLMs), with their rich world knowledge and strong semantic reasoning capabilities, offer new possibilities for addressing this task. However, existing approaches typically leverage VLMs to extract visual features and couple them with traditional object detectors, resulting in multi-stage error propagation that hinders perception accuracy. To overcome this limitation, we propose VLM-3D, the first end-to-end framework that enables VLMs to perform 3D geometric perception in autonomous driving scenarios. VLM-3D incorporates Low-Rank Adaptation (LoRA) to efficiently adapt VLMs to driving tasks with minimal computational overhead, and introduces a joint semantic-geometric loss design: token-level semantic loss is applied during early training to ensure stable convergence, while 3D IoU loss is introduced in later stages to refine the accuracy of 3D bounding box predictions. Evaluations on the nuScenes dataset demonstrate that the proposed joint semantic-geometric loss in VLM-3D leads to a 12.8% improvement in perception accuracy, fully validating the effectiveness and advancement of our method.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping

    cs.CV 2026-07 unverdicted novelty 5.0 of 10

    OmniView-Space framework with MPSM, tool-guided reasoning, and distillation achieves SOTA on spatial reasoning benchmarks for MLLMs while reducing external geometry dependencies.

Reference graph

Works this paper leans on

55 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Leng Cai, Junxuan He, Yikai Li, Junjie Liang, Yuanping Lin, Ziming Quan, Yawen Zeng, and Jin Xu. 2025. RTBAgent: A LLM-based Agent System for Real-Time Bidding. arXiv:2502.00792 [cs.AI] https://arxiv.org/abs/2502.00792

    work page Pith review arXiv 2025

  2. [2]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. 2024. Gui-world: A dataset for gui-oriented multimodal llm-based agents. arXiv e-prints (2024), arXiv–2406

    work page 2024

  3. [3]

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al . 2024. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NeurIPS 2024 Workshop on Open-World Agents

    work page 2024

  4. [4]

    Peng Chen, Pi Bu, Yingyao Wang, Xinyi Wang, Ziming Wang, Jie Guo, Yingxiu Zhao, Qi Zhu, Jun Song, Siran Yang, Jiamang Wang, and Bo Zheng. 2025. Combat- VLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Games. arXiv:2503.09527 [cs.CV] https://arxiv.org/abs/2503.09527

    arXiv 2025

  5. [5]

    Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan. 2018. Visual grounding via accumulated attention. In Proceedings of the IEEE conference on computer vision and pattern recognition . 7746–7755

    work page 2018

  6. [6]

    Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al. 2024. Mobile-bench: An evaluation benchmark for llm-based mobile agents. arXiv preprint arXiv:2407.00993 (2024)

    arXiv 2024

  7. [7]

    Difei Gao, Siyuan Hu, Zechen Bai, Qinghong Lin, and Mike Zheng Shou. 2024. AssistEditor: Multi-Agent Collaboration for GUI Workflow Automation in Video Creation. In Proceedings of the 32nd ACM International Conference on Multimedia . 11255–11257

    work page 2024

  8. [8]

    Jihao Gu, Qihang Ai, Yingyao Wang, Pi Bu, Jingxuan Xing, Zekun Zhu, Wei Jiang, Ziming Wang, Yingxiu Zhao, Ming-Liang Zhang, Jun Song, Yuning Jiang, and Bo Zheng. 2025. Mobile-R1: Towards Interactive Reinforcement Learning for VLM-Based Mobile Agent via Task-Level Rewards. arXiv:2506.20332 [cs.AI] https://arxiv.org/abs/2506.20332

    arXiv 2025

Show all 55 references
  1. [9]

    Yanheng He, Jiahe Jin, Shijie Xia, Jiadi Su, Runze Fan, Haoyang Zou, Xiangkun Hu, and Pengfei Liu. 2024. PC Agent: While You Sleep, AI Works–A Cognitive Journey into Digital World. arXiv preprint arXiv:2412.17589 (2024)

    2024 arXiv

  2. [10]

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al . 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352 3, 4 (2023), 6

    2023 arXiv

  3. [11]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14281–14290

    2024

  4. [12]

    Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, and Chi Zhang. 2025. AppAgentX: Evolving GUI Agents as Proficient Smartphone Users.arXiv preprint arXiv:2503.02268 (2025)

    2025 arXiv

  5. [13]

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. 2024. AutoWebGLM: A Large Language Model-based Web Navigating Agent. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and...

    2024

  6. [14]

    Juyong Lee, Dongyoon Hahm, June Suk Choi, W Bradley Knox, and Kimin Lee

  7. [15]

    Xiangyu Li, Yawen Zeng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. 2025. HedgeAgents: A Balanced-aware Multi-agent Financial Trading System. arXiv:2502.13165 [cs.MA] https://arxiv.org/abs/2502.13165

    2025 arXiv

  8. [16]

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping natural language instructions to mobile UI action sequences. arXiv preprint arXiv:2005.03776 (2020)

    2020 arXiv

  9. [17]

    Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. 2024. Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408.11824 (2024)

    2024

  10. [18]

    Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, et al. 2025. PC- Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC. arXiv preprint arXiv:2502.14282 (2025)

    2025 arXiv

  11. [19]

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Han- lin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. 2024. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820 (2024)

    2024 arXiv

  12. [20]

    Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to-structure generation for end-to-end event extraction. arXiv preprint arXiv:2106.09232 (2021)

    2021 arXiv

  13. [21]

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203 (2024)

    2024 arXiv

  14. [22]

    Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. Coco-agent: A compre- hensive cognitive mllm agent for smartphone gui automation. arXiv preprint arXiv:2402.11941 (2024)

    2024 arXiv

  15. [23]

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. 2025. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326 (2025)

    2025 arXiv

  16. [24]

    Kaixin Shen, Ruijie Quan, Linchao Zhu, Jun Xiao, and Yi Yang. 2024. Neural Interaction Energy for Multi-Agent Trajectory Prediction. In Proceedings of the 32nd ACM International Conference on Multimedia . 1952–1960

    2024

  17. [25]

    Yunpeng Song, Yiheng Bian, Yongtao Tang, Guiyu Ma, and Zhongmin Cai. 2024. Visiontasker: Mobile task automation using vision based ui understanding and llm task planning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–17

    2024

  18. [26]

    Jiahui Sun, Zhichao Hua, and Yubin Xia. 2025. AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents. arXiv preprint arXiv:2503.02403 (2025)

    2025

  19. [27]

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. Meta-gui: Towards multi-modal conversational agents on mobile gui. arXiv preprint arXiv:2205.11029 (2022)

    2022 arXiv

  20. [28]

    Qwen Team. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  21. [29]

    Qwen Team. 2025. Qwen2.5-VL Technical Report. arXiv:2502.13923 [cs.CV] https://arxiv.org/abs/2502.13923

    2025 arXiv

  22. [30]

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. arXiv preprint arXiv:2406.01014 (2024)

    2024 arXiv

  23. [31]

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158 (2024)

    2024 arXiv

  24. [32]

    Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. 2024. MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents.arXiv preprint arXiv:2406.08184 (2024)

    2024 arXiv

  25. [33]

    Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. 2025. Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks. arXiv preprint arXiv:2501.11733 (2025)

    2025 arXiv

  26. [34]

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Con- ference on Mobile Computing and Network...

    2024

  27. [35]

    Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, et al. 2024. AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation. arXiv preprint arXiv:2412.18116 (2024)

    2024 arXiv

  28. [36]

    Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. 2023. Droidbot-gpt: Gpt-powered ui automation for android. arXiv preprint arXiv:2304.07061 (2023)

    2023 arXiv

  29. [37]

    Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. 2024. Understanding the weakness of large language model agents within a complex android environment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 6061–6072

    2024

  30. [38]

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. 2024. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024 (2024)

    2024 arXiv

  31. [39]

    Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224 (2023)

    2023 arXiv

  32. [40]

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision . Springer, 240–255

    2024

  33. [41]

    Yawen Zeng, Da Cao, Xiaochi Wei, Meng Liu, Zhou Zhao, and Zheng Qin. 2021. Multi-modal relational graph for cross-modal video moment retrieval. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2215–2224

    2021

  34. [42]

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. AppAgent: Multimodal Agents as Smartphone Users. arXiv:2312.13771 [cs.CV]

    2023 arXiv

  35. [43]

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713 (2024)

    2024 arXiv

  36. [44]

    Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. 2024. Llamatouch: A faithful and scalable testbed for mobile ui task automation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology . 1–13

    2024

  37. [45]

    arrange a romantic dinner

    Kangjia Zhao, Jiahui Song, Leigang Sha, HaoZhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, and Jianwei Yin. 2024. GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent. arXiv preprint arXiv:2412.18426 (2024). �� ���� ����� ������� ������� ���� ���...

    2024 arXiv

  38. [47]

    Each instruction must explicitly mention the names of at least two mobile applications

  39. [48]

    Vague Instruction You are an expert in mobile agent instruction generation

    The number of applications mentioned must align with the nature of the task. Vague Instruction You are an expert in mobile agent instruction generation. Please generate 20 vague or underspecified task instruc- tions 10 in English and 10 in Chinese, that reflect ambigu- ous or ...

  40. [49]

    The instruction should not reference any specific mobile application

  41. [50]

    I’m hungry

    Ambiguous requests such as "I’m hungry" or "Make me pretty". Single-Application Instruction You are an expert in mobile agent instruction generation and have comprehensive knowledge of mainstream mo- bile applications. Please generate 20 mobile agent task instructions, 10 in E...

  42. [51]

    Each instruction must explicitly mention the name of one specific mobile application

  43. [52]

    The instruction must involve only one application. ������������ ������������ ������ ������ ��� ���������� ����� �� ���������� ������ ������������ ���������� ��� ��������� ������������ �� ���� ����� ������� ������� Interactive Instruction You are an expert in mobile agent instr...

  44. [53]

    Use user- name and password to log in to Google

    Personal information requirements, (e.g., "Use user- name and password to log in to Google"). Unethical Instruction You are an expert in mobile agent instruction generation and have comprehensive knowledge of mainstream mobile applications. Please generate 20 mobile agent task...

  45. [54]

    Search some words of racial discrimination

    Offensive content like "Search some words of racial discrimination"

  46. [55]

    Find pirated movies

    Negative behaviors such as "Find pirated movies". C.2 Model Training We fine-tune the Qwen2.5-VL-3B-Instruct model using LoRA. Train- ing is conducted for 3 epochs on 1000 samples with a maximum sequence length of 2048. We use the AdamW optimizer with a co- sine scheduler, a l...

  47. [2024]

    arXiv preprint arXiv:2410.17520 (2024)

    Mobilesafetybench: Evaluating safety of autonomous agents in mobile device control. arXiv preprint arXiv:2410.17520 (2024)

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.