Pith. sign in

REVIEW 5 major objections 5 minor 32 references

AndroidReality: How Far Are Mobile Agents from the Real World?

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mobile agents that score well on clean Android benchmarks lose 13 to 36 percentage points in success rate when realistic interface variations are injected, a gap this paper attributes to four recurring failure modes and partially closes…

desk verdict Useful perturbation taxonomy and benchmark with a credible qualitative brittleness story, but the headline numbers rest on single runs and the recovery mechanism is only demonstrated in-sample on one model. read the letter →

arxiv 2608.07775 v1 pith:4BBFRPON submitted 2026-08-07 cs.AI

Xiaoou Liu , Longchao Da , Hanyang Chen , Yuan Ling , Hua Wei This is my paper · ORCID
classification cs.AI
keywords mobileagentsrobustnessevaluationperturbationbenchmarkMarkovdecisionprocesstest-timeintrospectionGUIAndroidemulatorerroranalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mobile agents score well on clean online benchmarks, but that success may be misleading. This paper establishes robustness as a separate evaluation dimension by wrapping a standard Android task benchmark with a perturbation layer that injects realistic interface variability along three axes: the observed state (theme, font size, orientation, density), the transition (pop-ups, notifications, consent sheets), and the action (delays, drops, freezes). Under these perturbations, all eight tested agents lose 13 to 36 percentage points in overall success rate, with state-level visual changes causing the most uniform damage. A manual failure analysis attributes most failures to four recurring patterns, and a training-free Test-Time Introspective Recovery module that checks for these patterns at each step reduces failure counts by 17 percent on perturbed tasks and 20 percent on clean tasks. The upshot is that benchmark perturbation can expose latent weaknesses that clean evaluation hides, and that some of them are addressable without retraining.

What carries the argument

The load-bearing mechanism is the perturbation wrapper defined in equations (1)-(2): it takes the canonical MDP $M_0$ and applies operators $\phi^S_\theta$, $\phi^A_\theta$, and $\tilde{T}_\theta$ to produce a perturbed MDP that shares reward and task semantics, yielding a taxonomy (state, transition, action) that allows failure attribution by MDP component. The second mechanism is the TTIR diagnose-then-recover loop: a diagnoser that inspects the screenshot plus action history for four signals (stuck loop, unsatisfied action effect, interference, task misalignment) and a recovery module that dispatches a targeted corrective action before the agent continues.

What would settle it

Collect a corpus of naturally occurring real-world Android sessions that include the same variations (dark mode, pop-ups, loading delays, dropped taps) and run the same eight agents on those sessions; if the real-world success-rate drop is much smaller or larger than the 13-to-36-point drop measured under AndroidReality, the benchmark's estimate of the deployment gap is wrong. A cheaper check: rerun the benchmark with multiple seeds and repeated trials to see whether the per-category drops are stable or within noise.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that current mobile agents solve clean Android tasks well but are brittle to interface variations that are ordinary in real use: a perturbation wrapper that keeps the task goal and success verifier fixed while altering the observation, the transition, or the action execution converts the canonical environment into a family of perturbed Markov decision processes, and every evaluated agent's success rate falls substantially, by 13 to 36 points. The degradation is not random: state-level perturbations that change visual layout produce the largest and most uniform drops, and the failures cluster into four categories — stale-coordinate stuck loops, silent action failures, off-task engagement with distractions, and goal-memory drift after a distraction is dismissed. The same four categories also occur in clean settings, which the paper reads as evidence that these are latent weaknesses rather than artifacts of perturbation. Finally, the paper shows that a training-free, model-agnostic introspection loop that detects the four failure signs and launches targeted recovery actions reduces failed tasks on both perturbed and clean settings, without modifying the agent's underlying policy.

Load-bearing premise

The central assumption is that the injected perturbations faithfully reproduce the range of ordinary real-world interface variability while preserving the task's meaning; if the injected glitches are unrepresentative or accidentally change what counts as success, the measured drops do not tell us how far agents are from the real world.

Editorial extensions

If this is right

  • Success-rate comparisons on clean benchmarks should be supplemented by a robustness score under controlled perturbations; the paper's taxonomy provides a way to attribute failures to a specific MDP component.
  • Simply scaling models or adding post-training does not close the robustness gap: losses are comparable across model families and sizes from 2B to 32B.
  • State-level visual perturbations (theme, density, orientation) are the most uniformly damaging, so visual re-grounding is a priority target.
  • Even agents that correctly dismiss a pop-up can lose the original task goal, implying that memory of the goal matters as much as distraction detection.
  • A training-free introspection loop can recover a non-trivial fraction of failures on both perturbed and clean tasks, suggesting that explicit verification is a low-cost intervention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the realism assumption holds, the same perturbation methodology transfers to other GUI settings (web, desktop) and other task distributions, where the taxonomy may surface different recurring failure categories.
  • TTIR's gains on clean tasks hint that the four failure categories are a general diagnostic vocabulary for agent behaviour, not a perturbation-specific fix; a natural extension would be to train agents to internalize the verification and re-grounding steps so the recovery costs fewer extra steps.
  • The step overhead and budget effects imply that robustness should be measured not only in success rate but in step efficiency; the paper's delta-step metric invites a cost-benefit analysis of recovery strategies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces AndroidReality, a perturbation-based evaluation framework built on top of AndroidWorld. It organizes real-world interface variability into a taxonomy of state-, transition-, and action-level perturbations grounded in an MDP formulation. The authors evaluate eight open-source mobile agents under these perturbations, report substantial success-rate drops, and identify four recurring failure categories. They then propose Test-Time Introspective Recovery (TTIR), a training-free, model-agnostic mechanism that diagnoses these failure modes and applies targeted recovery strategies, and report that it reduces failed tasks in both perturbed and clean settings. The paper argues that robustness should be a standard dimension in mobile-agent evaluation and that benchmark perturbation can surface latent weaknesses.

Significance. If the quantitative claims are reliable, AndroidReality addresses a genuine gap: existing mobile-agent benchmarks are clean and stationary, whereas deployment requires coping with interface variability. The MDP-grounded taxonomy is principled, the perturbation configurations are parameterized and reproducible (fixed seeds, shared task-perturbation pairs, released code), and the four error categories are well illustrated with concrete trajectory examples. TTIR is training-free and model-agnostic, and the observation that it also helps in the clean setting is intriguing. However, the headline robustness and recovery numbers are not statistically anchored: results come from single runs without error bars, UI-TARS clean baselines are taken from external papers rather than rerun, and TTIR is evaluated in-sample on the same benchmark used to develop its categories. The 'real world' framing additionally rests on an unvalidated assumption that the synthetic perturbations faithfully emulate real-world variability. With stronger experimental grounding, the benchmark and taxonomy would be a useful contribution to the community.

major comments (5)
  1. [Section 4.2, Table 2] The claim that 'every evaluated agent suffers substantial degradation under perturbation, with absolute drops of 13–36 points on overall success rate' is contradicted by the table itself: UI-TARS-7B-SFT drops from 33.00 to 25.86, a 7.14-point drop, while UI-TARS-1.5-7B drops 36.61 points. The actual range across the table is 7.14 to 36.61, not 13 to 36, and the statement that degradation is 'uniformly across model families' is not supported by this exception. Because this range appears in the central observation of the paper, the text should be corrected or qualified to reflect the data.
  2. [Section 4.1 and Appendix B (execution protocol)] Appendix B states that 'Each task–perturbation pair is evaluated once under the fixed seed.' All headline numbers in Sections 4.2 and 5.2 are therefore single-run success rates with no confidence intervals or significance tests. Task success is binary, per-category subsets can be as small as a handful of tasks (see Table 5 and Figures 6–7, where several panels show denominators of roughly 5–15 tasks), and VLM sampling temperature is 0.1, so sampling noise alone can produce success-rate shifts of the reported magnitude. Please report repeated trials or binomial confidence intervals, and explicitly state the number of tasks behind each perturbation profile.
  3. [Table 2 (UI-TARS rows)] The clean success rates for UI-TARS-7B-SFT and UI-TARS-1.5-7B are marked with a dagger indicating they are taken from the original paper, while the perturbed numbers are produced by the authors' harness. Since the clean baselines were not rerun under the same serving conditions, perturbation wrapper, and evaluation protocol, the computed drops (e.g., -36.61 for UI-TARS-1.5-7B) conflate environment differences with perturbation effects. Please rerun these models on clean AndroidWorld in the same harness, or explicitly label the comparison as cross-paper and avoid making absolute drop claims for these rows.
  4. [Section 5.2, Figure 4] TTIR is evaluated on a single model (GUI-Owl-7B), with no repeated trials, no significance test, and no comparison against a simpler baseline (e.g., a generic 'try a different action' heuristic or a random re-grounding strategy). Moreover, the four error categories and TTIR's detection/recovery strategies were derived from the same AndroidReality trajectories on which TTIR is evaluated (Section 4.3 feeds into Section 5.1), so the reported improvements from 60 to 50 and 44 to 35 failed tasks are in-sample. Please evaluate on held-out perturbation types or a second benchmark, and compare against a non-introspective recovery baseline, to support the claim that TTIR recovers failures rather than overfitting the diagnosed categories.
  5. [Section 3.1 (Eq. 1–2) and title] The 'real world' conclusion rests on the assumption that the injected perturbations faithfully emulate real-world interface variability while preserving task semantics and reward. The paper provides no validation of this assumption, such as comparison with real device logs, deployment telemetry, crowdsourced real-world interaction data, or user studies. The results support the conclusion that agents are brittle to these specific controlled perturbations, but not directly the claim that they are 'far from the real world.' Please add validation tying the perturbation distributions to measured real-world conditions, or soften the real-world claims in the title and abstract.
minor comments (5)
  1. [Section 6 (first sentence)] The sentence 'we constructed AndroidReality, it is an online mobile benchmark' should be revised to 'we constructed AndroidReality, an online mobile benchmark' (remove 'it is').
  2. [Section 5.1 vs Appendix D.1] The default stuck-loop detection threshold is inconsistent: Section 5.1 states 'the last N steps (default N=3)', while Appendix D.1 says 'the last N steps (default N=5)' for the deterministic hard-stuck rule. Please harmonize these values or explain the discrepancy.
  3. [Figures 6 and 7 captions] The captions use 'Clean-before' and 'Perturbed-after' but it would be clearer to state explicitly that the clean condition is the same task subset without the wrapper, and the perturbed condition applies one perturbation profile per task.
  4. [Table 3] For the transition-level perturbations, the 'Tunable Parameters' column lists generic entries such as 'Surface style', 'Blocking', and 'Auto-dismiss duration' without default values; providing defaults or a pointer to Table 4 would improve reproducibility.
  5. [Section 4.2] The sentence 'The top-three challenging types are in Table 5' appears before Table 5 is referenced; consider adding a forward pointer or reordering the text so the reader can locate the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the degradation results and TTIR reductions are measured outcomes, not consequences of the paper's definitions or self-citations.

full rationale

We walked the claimed derivation chain. The MDP perturbation operator (Eqs. 1-2) defines what counts as a state, transition, or action perturbation, but the robustness drops in Table 2 are empirical measurements produced by AndroidWorld's programmatic verifier on actual agent runs; they are not entailed by the taxonomy. The four error categories in Section 4.3 are manual labels over logged trajectories, and they are not used as the success criterion: TTIR's recovery actions can fail, and final outcomes are judged by the external AndroidWorld verifier, so the 60-to-50 and 44-to-35 reductions in Section 5.2 are not forced by construction even though the strategies were motivated by the same benchmark's error analysis. The only self-citations (e.g., refs. [15] and [18]) are contextual surveys and do not carry a load-bearing uniqueness or derivation step. We also weighed the in-sample nature of TTIR and Appendix B's single-run protocol ('Each task-perturbation pair is evaluated once under the fixed seed'); these are external-validity and statistical-reliability concerns, not circularity, so they do not raise the circularity score.

Assumptions & free parameters 7 free parameters · 7 assumptions · 0 invented entities

The paper introduces a benchmark artifact and a recovery wrapper, but no new physical or conceptual entities. The main burden falls on the realism and evaluation-stability assumptions above.

free parameters (7)
  • FONT_SIZE_LARGE_SCALE = 1.35
    Hand-selected severity for the font_size_large perturbation (Table 4); no sensitivity analysis or real-world calibration.
  • FONT_SIZE_SMALL_SCALE = 0.82
    Hand-selected severity for font_size_small (Table 4).
  • DISPLAY_COMPACT_DENSITY_SCALE = 0.85
    Hand-selected display density for display_size_dpi_compact (Table 4).
  • ACTION_DELAY_PROBABILITY = 0.15
    Hand-set probability of delaying an action (Table 4); affects all action-delay results.
  • ACTION_DROP_PROBABILITY = 0.08
    Hand-set probability of dropping an action (Table 4).
  • Step budget multiplier = 20x human reference length
    Budget chosen for both clean and perturbed runs (Appendix B); a different budget changes failure rates, especially for transition perturbations.
  • TTIR stuck-loop threshold N = 3 (Section 5.1) or 5 (Appendix D.1)
    Trigger threshold for stuck-loop recovery; the paper is internally inconsistent about its value, and it is hand-set.
assumptions (7)
  • standard math MDP formalism with components (S,A,T,R,gamma)
    Section 3.1 defines M0 and perturbed MDPs; this is standard modeling.
  • domain assumption The injected perturbations preserve original task semantics and reward
    Section 3.1 states this assumption directly; it is not independently verified.
  • domain assumption The perturbation set is representative of real-world interface variability
    Introduction and Figure 1 claim realism, but no real-world data validates the selection or severity of perturbations.
  • domain assumption AndroidWorld's programmatic success verifier is a valid task-completion signal
    Section 3.2/3.3 relies on the unchanged verifier without re-validating it.
  • domain assumption A single run per task with fixed seed yields stable success-rate estimates
    Appendix B says each task-perturbation pair is evaluated once; no repeated runs or confidence intervals are reported.
  • domain assumption Manual error tagging of trajectories is consistent
    Section 4.3 describes manual tagging; no inter-annotator agreement is reported.
  • domain assumption The backbone VLM can serve as diagnoser and recovery agent without degrading the main agent
    Section 5.1 reuses the same VLM for TTIR; the possibility of interference with the main agent is not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AndroidReality: How Far Are Mobile Agents from the Real World?." pith.science (2026). https://pith.science/paper/4BBFRPON

@misc{pith2026260807775,
  author       = {Pith},
  title        = {Pith review of: AndroidReality: How Far Are Mobile Agents from the Real World?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4BBFRPON}},
  note         = {Machine review of arXiv:2608.07775}
}
read the original abstract

Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions. In this work, we introduce AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents. Through a Markov Decision Process (MDP) perspective, we organize real-world interface variability into a principled taxonomy of perturbations along three axes: state, transition, and action. Guided by this taxonomy, we build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections, enabling systematic robustness evaluation of mobile agents. Our evaluation reveals substantial robustness gaps and four recurring error categories, motivating a simple training-free Test-Time Introspective Recovery (TTIR) mechanism that mitigates these failures on both perturbed and clean settings. Together, these results position robustness as a missing dimension in mobile agent evaluation and establish benchmark perturbation as an effective tool for both stress testing and surfacing latent weaknesses of mobile agents.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 8 canonical work pages

  1. [1]

    Os-kairos: Adaptive interaction for mllm-powered gui agents

    Pengzhou Cheng, Zheng Wu, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang, and Gongshen Liu. Os-kairos: Adaptive interaction for mllm-powered gui agents. InFindings of the Association for Computational Linguistics: ACL 2025, pages 6701–6725, 2025

    2025

  2. [2]

    Generalization in online reinforcement learning for mobile agents.arXiv preprint arXiv:2603.07432, 2026

    Li Gu, Zihuan Jiang, Zhixiang Chi, Huan Liu, Ziqiang Wang, Yuanhao Yu, Glen Berseth, and Yang Wang. Generalization in online reinforcement learning for mobile agents.arXiv preprint arXiv:2603.07432, 2026

    arXiv 2026

  3. [3]

    Mobileuse: A hierarchical reflection-driven gui agent for autonomous mobile operation.Advances in Neural Information Processing Systems, 38:40361–40388, 2026

    Ning Li, Xiangmou Qu, Jiamu Zhou, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuying Peng, Jun Wang, and Weinan Zhang. Mobileuse: A hierarchical reflection-driven gui agent for autonomous mobile operation.Advances in Neural Information Processing Systems, 38:40361–40388, 2026

    work page 2026

  4. [4]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Ya...

    arXiv 2025

  5. [5]

    Mobile-agent-v3: Fundamental agents for gui automation.arXiv preprint arXiv:2508.15144, 2025

    Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Fundamental agents for gui automation.arXiv preprint arXiv:2508.15144, 2025

    arXiv 2025

  6. [6]

    Mobile-agent-v3.5: Multi-platform fundamental gui agents

    Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, Zhiyuan Chen, Jitong Liao, Qi Zheng, Jiahui Zeng, Ze Xu, Shuai Bai, Junyang Lin, Jingren Zhou, and Ming Yan. Mobile-agent-v3.5: Multi-platform fundamental gui agents. arXiv preprint arXiv:2602.16855, 2026

    arXiv 2026

  7. [7]

    Spa-bench: A comprehensive benchmark for smartphone agent evaluation

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  8. [8]

    Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments.arXiv preprint arXiv:2512.19432, 2025

    Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, et al. Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments.arXiv preprint arXiv:2512.19432, 2025

    arXiv 2025

Show all 32 references
  1. [9]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. InThe Thirteenth International Confe...

    2025

  2. [10]

    Androidlab: Training and systematic benchmarking of android autonomous agents

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. InProceedings of the 63rd Annual Meeting of the Association for Computational Ling...

    2025

  3. [11]

    Foundations and recent trends in multimodal mobile agents: A survey.arXiv preprint arXiv:2411.02006, 2024

    Biao Wu, Yanda Li, Zhiwei Zhang, Yunchao Wei, Meng Fang, and Ling Chen. Foundations and recent trends in multimodal mobile agents: A survey.arXiv preprint arXiv:2411.02006, 2024. 12 AndroidReality: How Far Are Mobile Agents from the Real World?

    2024 arXiv

  4. [12]

    Gui-robust: A comprehensive dataset for testing gui agent robustness in real-world anomalies

    Jingqi Yang, Zhilong Song, Jiawei Chen, Mingli Song, Sheng Zhou, Linjun Sun, Xiaogang Ouyang, Chun Chen, and Can Wang. Gui-robust: A comprehensive dataset for testing gui agent robustness in real-world anomalies. InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Dis...

    2026

  5. [13]

    D-gara: A dynamic benchmarking framework for gui agent robustness in real-world anomalies

    Sen Chen, Tong Zhao, Yi Bin, Fei Ma, Wenqi Shao, and Zheng Wang. D-gara: A dynamic benchmarking framework for gui agent robustness in real-world anomalies. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 17419–17426, 2026

    2026

  6. [14]

    Hyperclick: Advancing reliable gui grounding via uncertainty calibration

    Shaojie Zhang, Pei Fu, Ruoceng Zhang, Jiahui Yang, Anan Du, Xiuwen Xi, Shaokang Wang, Ying Huang, Bin Qin, Zhenbo Luo, et al. Hyperclick: Advancing reliable gui grounding via uncertainty calibration. arXiv preprint arXiv:2510.27266, 2025

    arXiv

  7. [15]

    A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models.arXiv preprint arXiv:2502.13187, 2025

    Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models.arXiv preprint arXiv:2502.13187, 2025

    2025 arXiv

  8. [16]

    A survey: Learning embodied intelligence from physical simulators and world models.arXiv preprint arXiv:2507.00917, 2025

    Xiaoxiao Long, Qingrui Zhao, Kaiwen Zhang, Zihao Zhang, Dingrui Wang, Yumeng Liu, Zhengjie Shu, Yi Lu, Shouzheng Wang, Xinzhe Wei, et al. A survey: Learning embodied intelligence from physical simulators and world models.arXiv preprint arXiv:2507.00917, 2025

    2025 arXiv

  9. [17]

    On the reliability of computer use agents.arXiv preprint arXiv:2604.17849, 2026

    Gonzalo Gonzalez-Pumariega, Saaket Agashe, Jiachen Yang, Ang Li, and Xin Eric Wang. On the reliability of computer use agents.arXiv preprint arXiv:2604.17849, 2026

    2026 arXiv

  10. [18]

    Lost in execution: On the multilingual robustness of tool calling in large language models.arXiv preprint arXiv:2601.05366, 2026

    Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani, Wanpeng Xu, Hua Wei, and Xiyang Hu. Lost in execution: On the multilingual robustness of tool calling in large language models.arXiv preprint arXiv:2601.05366, 2026

    2026 arXiv

  11. [19]

    Paligemma 2: A family of versatile vlms for transfer.arXiv preprint arXiv:2412.03555, 2024

    Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: A family of versatile vlms for transfer.arXiv preprint arXiv:2412.03555, 2024

    2024 arXiv

  12. [20]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  13. [21]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  14. [22]

    Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    2025 arXiv

  15. [23]

    Deepseek-ocr 2: Visual causal flow.arXiv preprint arXiv:2601.20552, 2026

    Haoran Wei, Yaofeng Sun, and Yukun Li. Deepseek-ocr 2: Visual causal flow.arXiv preprint arXiv:2601.20552, 2026

    2026

  16. [24]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. 2024. 13 AndroidReality: How Far Are Mobile Agents from the Real World?

    2024

  17. [25]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14281–14290, 2024

    2024

  18. [26]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025

    2025

  19. [27]

    Hijacking jarvis: Benchmarking mobile gui agents against unprivileged third parties

    Guohong Liu, Jialei Ye, Jiacheng Liu, Yuanchun Li, Wei Liu, Pengzhi Gao, Jian Luan, and Yunxin Liu. Hijacking jarvis: Benchmarking mobile gui agents against unprivileged third parties. InProceedings of the 2nd International Workshop on Edge and Mobile Foundation Models, pages ...

    2025

  20. [28]

    Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation.arXiv preprint arXiv:2604.08455, 2026

    Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, et al. Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation.arXiv preprint arXiv:2604.08455, 2026

    2026 arXiv

  21. [29]

    Androidinthewild: A large-scale dataset for android device control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. volume 36, pages 59708–59728, 2023

    2023

  22. [30]

    On the effects of data scale on ui control agents.Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents.Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    2024

  23. [31]

    Mobilesafetybench: Evaluating safety of autonomous agents in mobile device control

    Juyong Lee, Dongyoon Hahm, June Suk Choi, W Bradley Knox, and Kimin Lee. Mobilesafetybench: Evaluating safety of autonomous agents in mobile device control. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37565–37573, 2026

    2026

  24. [32]

    Record an audio clip using Audio Recorder app and save it

    Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents.arXiv preprint arXiv:2410.06703, 2024. 14 AndroidReality: How Far Are Mobile Agents from the Real World? A.Android...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.