Pith. sign in

REVIEW 3 major objections 7 minor 37 references

Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 7,138-scenario benchmark shows smartphone agents fail to warn users about most privacy-sensitive operations.

desk verdict A substantial human-checked privacy benchmark for phone agents, but the RA judge and PRR denominator need work before the exact scores can be trusted. read the letter →

arxiv 2508.19493 v2 pith:4GC7X2DO submitted 2025-08-27 cs.CR cs.CV

Zhixin Lin , Jungang Li , Shidong Pan , Yibo Shi , Yue Yao , Dongliang Xu This is my paper · ORCID
classification cs.CRcs.CV
keywords SAPA-BenchprivacyawarenesssmartphoneagentsmultimodallargelanguagemodelsleakageriskGUIprivacy-sensitivescenarios
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contemporary smartphone agents that act on screen content are granted extensive access to personal data, and this paper asks whether they recognize and warn about privacy-sensitive operations before executing them. It introduces SAPA-Bench, a benchmark of 7,138 real-world scenarios sampled from popular apps, each annotated with privacy type, sensitivity level, exposure modality, and an expected warning response. It then defines five metrics—privacy recognition, localization, level awareness, category awareness, and risk awareness—and evaluates seven off-the-shelf agents and vision-language models. The central finding is that privacy awareness is markedly inadequate across the board: even the best-performing closed-source model reaches a risk-awareness score of only 67.14%, while open-source models score around 30% or lower on recognition and near-zero on granular classification. The paper concludes that dedicated privacy training, alignment, and evaluation are urgently needed, and that prompt hints can partly compensate but do not solve the problem.

What carries the argument

The benchmark's core object is SAPA-Bench, a dataset of 7,138 privacy-sensitive scenarios created by filtering roughly 80,000 screenshots from GUI Odyssey and OS-Atlas, auto-generating instruction-response pairs with GPT-4o, then verifying them through a two-pass human review. Each scenario carries annotations for privacy presence, exposure modality (screenshot or instruction), one of eight privacy categories, a three-tier sensitivity level, and an expected warning response. Five metrics—PRR, PLR, PLAR, PCAR, and RA—measure recognition, localization, severity estimation, category classification, and risk-aware response generation respectively. The RA metric uses an LLM judge to compare agent responses against reference risk prompts, while the other four compare against human-verified labels.

What would settle it

Re-score RA on a random subset using human judges who are blind to the reference risk prompts, then compare the resulting ranking with the paper's LLM-judge ranking; if models that score high under the LLM judge fall behind or vice versa, the RA metric is measuring protocol artifacts rather than privacy awareness. Alternatively, run a live deployment probe in which an agent is given a task like pasting a password and the actual behavior is observed; if an agent with high RA still executes without warning, the benchmark's claim about privacy awareness fails to predict behavior.

Watch

Extended reading notes

Core claim

Every tested agent fails to reliably detect, localize, classify, or warn about privacy leakage in smartphone tasks. Across 7,138 scenarios, all models score below 85% on Privacy Recognition Rate; open-source systems such as Show-UI, SpiritSight Agent, Qwen2.5-VL, and InternVL2.5 sit near 30%, meaning most sensitive scenarios go unnoticed. Fine-grained metrics are worse: privacy level awareness and category awareness hover in the single to low-double digits for most models, and risk awareness peaks at 67.14% even for the best model under explicit hints. Higher-sensitivity scenarios are more likely to be recognized than low- and medium-sensitivity ones, and closed-source models generally outperform open-source ones across all metrics. The paper interprets this as evidence that current agents optimize task completion without the privacy guardrails needed for safe deployment.

Load-bearing premise

The benchmark's measurements of privacy awareness assume that the GPT-4o-generated, human-verified ground truth and the LLM-based risk-awareness judge capture genuine privacy understanding rather than rewarding wording similar to the reference prompts.

Editorial extensions

If this is right

  • Agents currently lack the basic prerequisite for safe autonomy: they do not reliably know when an operation touches credentials, contacts, payment, or location.
  • Even the strongest commercial model falls short of deployment-grade privacy reliability, so off-the-shelf agents should not be entrusted with sensitive operations without an external privacy-check module.
  • Explicit prompt hints reliably improve risk awareness across models, so prompt-level interventions are a low-cost partial mitigation while training-based fixes are developed.
  • Low- and medium-sensitivity privacy events are the ones most often missed, meaning the largest privacy risk is not dramatic credential theft but ordinary contextual leakage such as pasted emails or shared meeting links.
  • The open-source and closed-source gap indicates that alignment data and post-training, not just base vision-language ability, drive privacy behavior; reproducing that alignment in open models is a concrete next target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The RA metric's LLM judge compares semantic alignment with a GPT-4o-written reference risk prompt, so scores may partly reward response style or template overlap rather than a genuine decision to halt or warn; re-scoring with human raters or a judge blind to the reference wording would test this.
  • The benchmark's instruction-response pairs are also GPT-4o-generated, so the dataset may inherit GPT-4o's privacy biases; a fully human-authored subset or a perturbation test with paraphrased instructions would reveal whether results change.
  • A natural extension is a live-environment probe: deploy an agent on a real device, inject a PIN-entry or clipboard-paste task, and measure whether the agent actually pauses and warns before acting, comparing against SAPA-Bench's predicted RA.
  • The finding that higher-sensitivity scenarios are easier to detect suggests a possible training curriculum: ordering synthetic privacy training data by sensitivity level could give open-source models a cheap alignment signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The manuscript introduces SAPA-Bench, a benchmark of 7,138 privacy-sensitive smartphone UI scenarios sampled from GUI-Odyssey and OS-Atlas, with annotations for privacy type, sensitivity level, leakage modality, and expected risk-warning responses. A five-stage pipeline combines GPT-4o-based automatic generation with human verification. Seven smartphone agents or generalist MLLMs are evaluated on five metrics (PRR, PLR, PLAR, PCAR, RA), and the paper reports that most agents have low privacy awareness, that open-source models trail closed-source models, that explicit hints improve risk awareness, and that higher-sensitivity scenarios are more detectable. The authors conclude that current smartphone agents lack adequate privacy safeguards and release code, data, and a project page.

Significance. If the findings hold, SAPA-Bench addresses a real gap: prior mobile-agent benchmarks measure task competence, not privacy awareness. The paper's strengths include the large scale (7,138 scenarios), the use of a two-pass human verification pipeline for ground-truth labels, the multi-dimensional metric design, and the public release of code and data. The qualitative direction of the findings—especially the weakness of open-source agents—is plausible and is supported by PRR, PLR, PLAR, and PCAR even if RA is set aside. However, the exact RA magnitudes and the Gemini-versus-GPT-4o ranking rest on an unvalidated LLM judge that shares a generator with the benchmark, and the PRR metric is computed without negative controls; these issues must be fixed before the quantitative claims can be relied upon.

major comments (3)
  1. [Abstract; Results and Discussion, Table 3] The abstract's headline claim that performance "remains below 60% even with explicit hints" is directly contradicted by Table 3, where Gemini 2.0-flash reaches 67.14% in the Explicit Hint condition, and by the main text's statement that "Gemini to 67.14%." The abstract should be revised to say "almost all models" or otherwise exempt Gemini, and the exact prompting condition should be stated.
  2. [Experiments, Evaluation Metrics; Annotation Pipeline, Stage 2] The RA metric relies on an LLM performing semantic alignment between the agent's response and a reference risk prompt, but those reference prompts were generated by GPT-4o in Stage 1, GPT-4o is itself one of the seven benchmarked models, and no human validation of the LLM judge is reported. Because Stage 2 forces ground-truth responses into a standardized "This action may result in ... Please confirm before proceeding" template, the judge may reward template-like phrasing rather than genuine privacy awareness. The exact RA scores and the Gemini-over-GPT-4o ranking are therefore not established by the current protocol; the authors should validate the judge against human ratings, report inter-rater agreement, and include reference-free or neutral-response baselines.
  3. [Experiments, Evaluation Metrics (PRR); Dataset Statistics] PRR is defined as the proportion of all samples that the agent flags as privacy-related, and every one of the 7,138 benchmark scenarios is privacy-sensitive by construction; there are no non-sensitive control scenarios. As a result, a model that always says "this is private" would receive 100% PRR, so the reported low PRR values cannot by themselves support the claim that "the vast majority of sensitive scenarios go undetected." To separate detection from over-flagging, the benchmark needs a set of non-sensitive or neutral scenarios and a false-positive or over-flagging statistic.
minor comments (7)
  1. [Experiments, Evaluation Metrics] "Evaluation Matrics" is a typo; it should be "Evaluation Metrics."
  2. [Table 1; Experiments] Model names are inconsistent across the paper: "LLaV A-NeXT" appears alongside "LLaVA-NeXT," and "InternVL 1.5" appears in one place while "InternVL2.5" is used elsewhere; please standardize.
  3. [Table 1] The column header layout for PLR (Image/Instruction/Overall) is ambiguous, and it is not immediately clear that the RA column in Table 1 corresponds to the Explicit Hint condition only; please clarify the header and state the condition explicitly.
  4. [Introduction; Abstract] The introduction says performance remains "below 60% even with implicit hints," while the abstract says "explicit hints"; these statements should be aligned, and the Gemini value should be corrected in both places.
  5. [Results and Discussion] The text reports "GPT-4o and Gemini score 55.03% and 66.14%, respectively," but Table 3 lists 67.14% for Gemini under the Explicit Hint condition; the inconsistency should be corrected.
  6. [Annotation Pipeline, Stage 4] The paper states that two annotators independently review each sample but reports no inter-annotator agreement statistic; adding Cohen's kappa or a similar measure for the privacy type, sensitivity level, and location labels would strengthen the claim of high-quality ground truth.
  7. [Results and Discussion; Tables 1-3] No confidence intervals or standard errors are reported for any of the five metrics; with 7,138 scenarios, bootstrap or analytic intervals would clarify which model differences are meaningful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark labels are human-verified and metrics compare model outputs to those labels, so no derivation reduces to its own inputs.

full rationale

The paper is an empirical benchmark rather than a derivation, and I could not exhibit any step in which a claimed result is equivalent to its own inputs by construction. The ground-truth annotations for privacy presence, type, sensitivity level, and location are initially generated by GPT-4o but then pass two stages of human verification (Stage 2 and Stage 4), so the labels used for PRR, PLR, PLAR, and PCAR are not model-fitted values. Those four metrics directly compare model outputs to these human-verified labels, which is standard benchmark measurement and not a circular reduction. The RA metric is defined as LLM-based semantic alignment to a reference risk prompt; although the reference prompts originate from GPT-4o and GPT-4o is one of the evaluated models, the paper does not specify that the RA judge is GPT-4o, the reference prompts are human-verified, and the RA score is not used to derive the other metrics. The possible non-independence of an unvalidated LLM judge raises a validity concern, but it is not a circularity: no equation forces a model's RA to equal the judge's output by construction, and no fitted parameter is renamed as a prediction. Self-citations in the paper (e.g., Pan et al. 2024 for FIPP) are contextual background and do not carry the argument. The internal inconsistency between the abstract's 'below 60% even with explicit hints' and Table 3's 67.14 for Gemini 2.0-flash is a correctness or consistency issue, not a circularity. Therefore, under the hard rule that circularity must be demonstrated by a specific reduction, the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on several domain assumptions about taxonomy validity, GPT-4o's reliability as an annotator, the LLM judge for RA, and the representativeness of the sampled scenarios. No numeric free parameters are fitted, but the evaluation protocol itself contains built-in assumptions that affect all reported scores.

assumptions (4)
  • domain assumption The eight privacy categories and three sensitivity levels, adapted from Apple and Google privacy labels and prior taxonomies, are a valid and complete decomposition of privacy leakage in smartphone interactions.
    Used throughout benchmark construction and metrics; if the taxonomy is wrong, PRR, PLAR, and PCAR scores do not measure privacy awareness.
  • domain assumption GPT-4o can reliably identify privacy-sensitive screenshots and generate realistic instruction-response pairs that human annotators can verify.
    Stage 0 and Stage 1 rely on GPT-4o for filtering and generation; only 400 screenshots are spot-checked and no filter precision or recall is reported.
  • domain assumption LLM-based semantic alignment between agent responses and reference risk prompts is a valid proxy for human judgment of risk awareness.
    RA is computed by an LLM aligner; the paper reports no human agreement or validation for this scoring procedure.
  • domain assumption The 7,138 sampled scenarios are representative of real privacy-sensitive operations encountered in smartphone use.
    Scenarios are sampled from GUI-Odyssey and OS-Atlas after GPT-4o filtering; no evidence is provided that the filtered distribution matches real-world privacy exposure frequencies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents." pith.science (2026). https://pith.science/paper/4GC7X2DO

@misc{pith2026250819493,
  author       = {Pith},
  title        = {Pith review of: Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GC7X2DO}},
  note         = {Machine review of arXiv:2508.19493}
}
read the original abstract

Smartphones bring significant convenience to users but also enable devices to extensively record various types of personal information. Existing smartphone agents powered by Multimodal Large Language Models (MLLMs) have achieved remarkable performance in automating different tasks. However, as the cost, these agents are granted substantial access to sensitive users' personal information during this operation. To gain a thorough understanding of the privacy awareness of these agents, we present the first large-scale benchmark encompassing 7,138 scenarios to the best of our knowledge. In addition, for privacy context in scenarios, we annotate its type (e.g., Account Credentials), sensitivity level, and location. We then carefully benchmark seven available mainstream smartphone agents. Our results demonstrate that almost all benchmarked agents show unsatisfying privacy awareness (RA), with performance remaining below 60% even with explicit hints. Overall, closed-source agents show better privacy ability than open-source ones, and Gemini 2.0-flash achieves the best, achieving an RA of 67%. We also find that the agents' privacy detection capability is highly related to scenario sensitivity level, i.e., the scenario with a higher sensitivity level is typically more identifiable. We hope the findings enlighten the research community to rethink the unbalanced utility-privacy tradeoff about smartphone agents. Our code and benchmark are available at https://zhixin-l.github.io/SAPA-Bench.

Figures

Figures reproduced from arXiv: 2508.19493 by the authors.

Figure 1
Figure 1. Motivation of SAPA-Bench. Left: Current agents [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Five-stage annotation pipeline of SAPA-Bench. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of SAPA-Bench, illustrating its task taxonomy, representative examples, and risk-level distribution. (a) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of performance across six evaluation [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Each model’s Risk Awareness (RA) under three [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies

    Ali, M. M.; Balash, D. G.; Kodwani, M.; Kanich, C.; and Aviv, A. J. 2024. Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies. arXiv:2306.17063

    work page Pith review arXiv 2024

  4. [5]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025 b . Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  5. [6]

    Chen, J.; Yuen, D.; Xie, B.; Yang, Y.; Chen, G.; Wu, Z.; Yixing, L.; Zhou, X.; Liu, W.; Wang, S.; et al. 2024 a . Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NeurIPS 2024 Workshop on Open-World Agents

    work page 2024

  6. [7]

    Chen, Z.; Wang, W.; Tian, H.; Ye, S.; Gao, Z.; Cui, E.; Tong, W.; Hu, K.; Luo, J.; Ma, Z.; Ma, J.; Wang, J.; Dong, X.; Yan, H.; Guo, H.; He, C.; Shi, B.; Jin, Z.; Xu, C.; Wang, B.; Wei, X.; Li, W.; Zhang, W.; Zhang, B.; Cai, P.; Wen, L.; Yan, X.; Dou, M.; Lu, L.; Zhu, X.; Lu, T.; Lin, D.; Qiao, Y.; Dai, J.; and Wang, W. 2024 b . How far are we to GPT-4V? ...

    work page 2024

  7. [8]

    Dai, G.; Jiang, S.; Cao, T.; Li, Y.; Yang, Y.; Tan, R.; Li, M.; and Qiu, L. 2025. Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment

    work page 2025

  8. [9]

    Dang, Y.; Gao, M.; Yan, Y.; Zou, X.; Gu, Y.; Liu, A.; and Hu, X. 2024. Exploring response uncertainty in mllms: An empirical evaluation under misleading scenarios. arXiv preprint arXiv:2411.02708

    arXiv 2024

Show all 37 references
  1. [10]

    Deng, S.; Xu, W.; Sun, H.; Liu, W.; Tan, T.; Liu, J.; Li, A.; Luan, J.; Wang, B.; Yan, R.; and Shang, S. 2024. Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Long Pa...

    2024

  2. [11]

    Huang, Z.; Cheng, Z.; Pan, J.; Hou, Z.; and Zhan, M. 2025. Spiritsight agent: Advanced gui agent with one look. In Proceedings of the Computer Vision and Pattern Recognition Conference, 29490--29500

    2025

  3. [12]

    P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al

    Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  4. [13]

    H.; Alaqra, A

    Iwaya, L. H.; Alaqra, A. S.; Hansen, M.; and Fischer-H \"u bner, S. 2024. Privacy Impact Assessments in the Wild: A Scoping Review. CoRR, abs/2402.11193

    2024 arXiv

  5. [14]

    T.; and Zhang, C

    Jiang, W.; Zhuang, Y.; Song, C.; Yang, X.; Zhou, J. T.; and Zhang, C. 2025. AppAgentX: Evolving GUI Agents as Proficient Smartphone Users

    2025

  6. [15]

    Khandelwal, R.; Nayak, A.; Chung, P.; and Fawaz, K. 2023. Comparing privacy labels of applications in android and iOS. In Proceedings of the 22nd Workshop on Privacy in the Electronic Society, 61--73

    2023

  7. [16]

    S.; Knox, W

    Lee, J.; Hahm, D.; Choi, J. S.; Knox, W. B.; and Lee, K. 2024. Mobilesafetybench: Evaluating safety of autonomous agents in mobile device control. arXiv preprint arXiv:2410.17520

    2024

  8. [17]

    Li, Q.; Hong, J.; Xie, C.; Tan, J.; Xin, R.; Hou, J.; Yin, X.; Wang, Z.; Hendrycks, D.; Wang, Z.; et al. 2024 a . Llm-pbe: Assessing data privacy in large language models. arXiv preprint arXiv:2408.12787

    2024 arXiv

  9. [18]

    Li, Y.; Zhang, C.; Yang, W.; Fu, B.; Cheng, P.; Chen, X.; Chen, L.; and Wei, Y. 2024 b . Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408.11824

    2024

  10. [19]

    Q.; Li, L.; Gao, D.; Yang, Z.; Wu, S.; Bai, Z.; Lei, S

    Lin, K. Q.; Li, L.; Gao, D.; Yang, Z.; Wu, S.; Bai, Z.; Lei, S. W.; Wang, L.; and Shou, M. Z. 2025. Showui: One vision-language-action model for gui visual agent. In Proceedings of the Computer Vision and Pattern Recognition Conference, 19498--19508

    2025

  11. [20]

    Liu, G.; Zhao, P.; Liu, L.; Guo, Y.; Xiao, H.; Lin, W.; Chai, Y.; Han, Y.; Ren, S.; Wang, H.; Liang, X.; Wang, W.; Wu, T.; Li, L.; Wang, H.; Xiong, G.; Liu, Y.; and Li, H. 2025. LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects. CoRR, abs/2504.19838

    2025

  12. [21]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

    2024

  13. [22]

    Lu, Q.; Shao, W.; Liu, Z.; Meng, F.; Li, B.; Chen, B.; Huang, S.; Zhang, K.; Qiao, Y.; and Luo, P. 2024. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451

    2024 arXiv

  14. [23]

    Ma, X.; Zhang, Z.; and Zhao, H. 2024. CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

    2024

  15. [24]

    Nema, P.; Anthonysamy, P.; Taft, N.; and Peddinti, S. T. 2022. Analyzing user perspectives on mobile app privacy at scale. In Proceedings of the 44th international conference on software engineering, 112--124

    2022

  16. [25]

    Pan, S.; Tao, Z.; Hoang, T.; Zhang, D.; Li, T.; Xing, Z.; Xu, X.; Staples, M.; Rakotoarivelo, T.; and Lo, D. 2024. A NEW HOPE : Contextual Privacy Policies for Mobile Applications and An Approach Toward Automated Generation. In 33rd USENIX Security Symposium (USENIX Security 2...

    2024

  17. [26]

    Rawles, C.; Li, A.; Rodriguez, D.; Riva, O.; and Lillicrap, T. 2023. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36: 59708--59728

    2023

  18. [27]

    Reid, M.; Savinov, N.; Teplyashin, D.; et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530

    2024 arXiv

  19. [28]

    K.; Choetkiertikul, M.; Ragkhitwetsagul, C.; and Ghose, A

    Sangaroonsilp, P.; Dam, H. K.; Choetkiertikul, M.; Ragkhitwetsagul, C.; and Ghose, A. 2023. A Taxonomy for Mining and Classifying Privacy Requirements in Issue Reports. Information and Software Technology, 157: 107162

    2023

  20. [29]

    Tang, F.; Xu, H.; Zhang, H.; Chen, S.; Wu, X.; Shen, Y.; Zhang, W.; Hou, G.; Tan, Z.; Yan, Y.; Song, K.; Shao, J.; Lu, W.; Xiao, J.; and Zhuang, Y. 2025. A Survey on (M)LLM-Based GUI Agents. CoRR, abs/2504.13865

    2025 arXiv

  21. [30]

    Wang, J.; Xu, H.; Ye, J.; Yan, M.; Shen, W.; Zhang, J.; Huang, F.; and Sang, J. 2024 a . Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158

    2024 arXiv

  22. [31]

    Wang, J.; Xu, H.; Zhang, X.; Yan, M.; Zhang, J.; Huang, F.; and Sang, J. 2025. Mobile-Agent-V: Learning Mobile Device Operation Through Video-Guided Multi-Agent Collaboration. CoRR, abs/2502.17110

    2025 arXiv

  23. [32]

    Wang, L.; Deng, Y.; Zha, Y.; Mao, G.; Wang, Q.; Min, T.; Chen, W.; and Chen, S. 2024 b . MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents. CoRR, abs/2406.08184

    2024 arXiv

  24. [33]

    Wang, S.; Ye, X.; Cheng, Q.; Duan, J.; Li, S.; Fu, J.; Qiu, X.; and Huang, X. 2024 c . Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model. arXiv preprint arXiv:2406.15279

    2024 arXiv

  25. [34]

    Wu, B.; Li, Y.; Fang, M.; Song, Z.; Zhang, Z.; Wei, Y.; and Chen, L. 2024 a . Foundations and Recent Trends in Multimodal Mobile Agents: A Survey. CoRR, abs/2411.02006

    2024 arXiv

  26. [35]

    P.; and Qiao, Y

    Wu, Z.; Wu, Z.; Xu, F.; Wang, Y.; Sun, Q.; Jia, C.; Cheng, K.; Ding, Z.; Chen, L.; Liang, P. P.; and Qiao, Y. 2024 b . OS-ATLAS: A Foundation Action Model for Generalist GUI Agents. CoRR, abs/2410.23218

    2024 arXiv

  27. [36]

    Xu, Y.; Liu, X.; Sun, X.; Cheng, S.; Yu, H.; Lai, H.; Zhang, S.; Zhang, D.; Tang, J.; and Dong, Y. 2025. AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (L...

    2025

  28. [37]

    Xun, S.; Tao, S.; Li, J.; Shi, Y.; Lin, Z.; Zhu, Z.; Yan, Y.; Li, H.; Zhang, L.; Wang, S.; et al. 2025. RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. arXiv preprint arXiv:2505.02064

    2025

  29. [38]

    Zhu, J.; Wang, W.; Chen, Z.; Liu, Z.; Ye, S.; Gu, L.; Tian, H.; Duan, Y.; Su, W.; Shao, J.; et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.