Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A backdoor that changes only pixels in training screenshots can silently hijack a vision-language mobile agent's actions and written explanations at inference time.

desk verdict VIBMA is the first clean-text visual backdoor for mobile agents with broad experiments, but the core gradient-alignment mechanism is under-validated and needs a control baseline. read the letter →

arxiv 2506.13205 v6 pith:CITOJAC4 submitted 2025-06-16 cs.CR cs.AI

Xuan Wang , Siyuan Liang , Zhe Liu , Yi Yu , Aishan Liu , Yuliang Lu , Xitong Gao , Ee-Chien Chang This is my paper
classification cs.CRcs.AI
keywords clean-textpoisoningvisualbackdoorattackvision-languagemodelsmobileagentsgradientalignmentdataUIautomationLoRAfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a mobile agent built from a vision-language model can be silently reprogrammed at training time by perturbing only the pixels of screenshots, leaving all text prompts and labels untouched. The attack, called VIBMA, plants a visual trigger in a small fraction of fine-tuning images; when that trigger later appears on screen, the agent executes an attacker-chosen action and produces an attacker-chosen rationalization, while behaving normally on clean screens. The authors argue this matters because mobile agents handle sensitive operations like messaging, calls, camera, and location with limited auditability, and existing safety checks focus on prompt text rather than visual integrity. Their experiments across three VLM backbones and six Android apps report attack success rates up to 94.67% with clean-task behavior largely preserved, making the case that visual-only poisoning is a practical and stealthy threat in mobile-agent adaptation pipelines.

What carries the argument

The central object is the gradient-alignment poisoning objective, Eq. (3): the attacker minimizes the cosine distance between the gradient of the loss at a target instance $(x_{\mathrm{target}}, t, y_{\mathrm{target}})$ (a clean screenshot carrying the visual trigger, paired with the attacker's desired action and rationale) and the average gradient over the poisoned samples. Poisoned images are clean screenshots plus small perturbations $\delta_i$ with $\|\delta_i\|_\infty \le \epsilon$, optimized with signed Adam, multiple restarts, and differentiable augmentation over the frozen backbone, then mixed into the fine-tuning set. The mismatch between the frozen-model optimization and the LoRA fine-tuning that actually implants the backdoor is the load-bearing assumption of the whole construction.

What would settle it

Use the authors' own poisoned dataset and optimization recipe but replace LoRA fine-tuning with full-model fine-tuning, or with a different LoRA rank and seed; if the action attack success rate drops to near the rate on clean triggered screenshots, the frozen-gradient-to-fine-tuning transfer is the load-bearing step, and the attack's claimed generality across adaptation schemes would be falsified.

Watch

Extended reading notes

Core claim

VIBMA is the first clean-text backdoor attack against VLM-based mobile agents. Using only imperceptible visual perturbations bounded by an $\ell^\infty$ budget, it injects a backdoor into a mobile agent during fine-tuning while keeping prompts, instructions, and labels completely unchanged. At inference, a predefined visual trigger (a static patch, a moving pattern, or a low-opacity blended object) forces the agent to emit an attacker-specified action-and-rationale pair, ranging from benign misactivation to privacy violation, malicious hijack, and policy shift. The attack is optimized by aligning the training gradients of poisoned samples with the gradient of a chosen target instance on a frozen pretrained VLM, so the backdoor embeds without any textual anomaly; the authors report action success up to 94.67% and follow-step ratio up to 95.85%, with resistance to resizing, JPEG compression, and cropping.

Load-bearing premise

The attack's success rests on the assumption that pixel changes chosen using a frozen pretrained model still work after the model is lightly fine-tuned on a mix of clean and poisoned data; the paper asserts rather than proves that this transfer holds.

Editorial extensions

If this is right

  • If VIBMA works as reported, any pipeline that fine-tunes a mobile agent on user-collected screenshots inherits a latent trigger that a single corrupted data source can activate across unseen apps and screens.
  • Because the backdoor survives fine-tuning and common corruptions like JPEG compression and resizing, defenses limited to input sanitization or prompt filtering will likely miss it.
  • The clean-data behavior loss is small (FSR drop of a few points), so standard accuracy-based validation on clean tasks would not flag the poisoned model.
  • The four attack types show the same poisoning mechanism can drive a range of harms, from benign misfires to policy-level behavioral shifts, meaning the threat spans more than single-action replacement.
  • The attack works across three different VLM backbones, suggesting the gradient-alignment recipe transfers across architectures rather than overfitting one model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same gradient-alignment recipe may transfer to other parameter-efficient adapters or to desktop and web agents that render graphical UIs, since the method only requires a frozen pretrained VLM and visual-trigger injection; this is a testable extension the paper does not run.
  • A practical defense direction suggested, but not tested, by the paper is monitoring the visual channel during adaptation: a surprisingly small number of screenshots whose gradients cluster around one target instance could be flagged before fine-tuning even starts.
  • The reported data efficiency (over 80% ASR at 10% poisoning) implies that even a modestly contaminated crowd-sourced dataset is a viable attack vector, raising the bar for what 'clean' training data must mean in agent pipelines.
  • Because the trigger is purely visual, the same attack could conceivably be activated by an adversary who merely renders a pattern on the victim's screen at inference time, without any text injection; the paper hints at but does not develop this escalation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VIBMA, a clean-text visual backdoor attack against VLM-based mobile agents. The attack poisons only the visual stream by adding small perturbations to images while leaving text prompts and labels unchanged, using a gradient-alignment objective that matches poisoned-sample gradients to a chosen target instance. At inference time, a separately added visual trigger (static patch, hoverball, or blended object) activates attacker-specified actions and rationales. Experiments on RICO, AITW, and crawled real-app screenshots across three VLM backbones report action success rates up to 94.67% and clean-task follow-step ratios up to 95.85%, along with ablations on poison ratio, perturbation budget, trigger size/position, and robustness to image corruptions.

Significance. If the central mechanism is verified, this is a timely and relevant contribution to the security of VLM-based mobile agents. The paper's strengths are its broad empirical coverage: six applications, three VLM backbones, three trigger designs, four attack types, and a reasonably detailed threat model. The ablations over poisoning ratio, perturbation budget, and trigger geometry provide useful practical guidance. However, the absence of mechanism-control experiments, the lack of variance reporting, and the reliance on synthetic RICO annotations mean the quantitative claims are currently stronger than the evidence supports. The paper does not ship machine-checked proofs or code, but for an empirical attack paper the main need is additional experimental control rather than formal derivation.

major comments (4)
  1. [Gradient-Aligned Poisoning Objective, Eq. (3), Algorithm 1, Section A.6] The poisoning objective aligns gradients of the frozen pre-trained model f_theta, but the subsequent fine-tuning uses LoRA, which updates only low-rank adapter parameters. The transfer of the frozen-model gradient alignment to the LoRA update direction is asserted rather than demonstrated. Because the novelty claim rests on this gradient-alignment mechanism, please provide (i) a measurement of the cosine similarity between target and poisoned-sample gradients before and after a few LoRA fine-tuning steps, or (ii) an ablation replacing the aligned perturbations with random perturbations of the same budget and trigger, and (iii) a trigger-only poisoning baseline without gradient alignment. If the random-perturbation baseline achieves comparable ASR, the high ASR could be explained by visible-trigger memorization rather than by the Eq. (3) objective.
  2. [Tables 1-5] All experimental tables report single point estimates with no variance, number of runs, or significance tests. For example, Table 1 reports Hurdle ASR of 91.05% versus Hoverball ASR of 86.26%, and the text concludes that Hurdle 'achieves the best balance' and that 'no trade-off is observed' between attack success and clean-task fidelity. Without error bars or repeated-seed statistics, these comparative and trade-off claims are not supported. Please report mean plus/minus standard deviation over at least three independent fine-tuning runs and apply an appropriate significance test for the key trigger and backbone comparisons.
  3. [Experiments, 'Main Results' and 'Ablation and Robustness Analysis'] The experimental section contains no comparison against prior backdoor or poisoning baselines. The related work cites BadAgent and Yang et al. for agent backdoors, but neither is evaluated here, and there is no baseline of simple visible-trigger poisoning without gradient alignment. To support the claim that VIBMA's gradient-aligned visual perturbation is necessary and that the attack improves over existing approaches, please add comparisons against at least a trigger-only poisoning baseline, a random-perturbation control, and one prior agent-poisoning method on a shared app/backbone configuration, or explicitly explain why existing baselines are not transferable to this setting.
  4. [Sections A.3 and A.4] The RICO dataset is converted into training and evaluation samples by generating prompts with GPT-4 and inferring actions by matching salient UI regions to prompt semantics; no human validation of these synthetic annotations is reported. Since ASR and FSR are computed against these labels, the RICO results in Tables 3 and 4 may reflect alignment with the label generator rather than with true user intent. Please provide a human-validated subset of RICO annotations, or report the main claims on AITW and the real-world crawled data alone, to rule out synthetic-label bias.
minor comments (5)
  1. [Abstract] The sentence 'These findings is the first to reveal the security vulnerabilities' is grammatically incorrect; it should read 'These findings are the first...' or 'This work is the first...'.
  2. [Figure 2] The x-axis label contains the corrupted token '/glyph1197avigation', which appears to be a rendering artifact; also, the text inconsistently refers to 'App Market' and 'App Store' for what appears to be the same application.
  3. [Table 3] The caption does not define the abbreviations A-ASR and C-ASR; please define them in the caption or refer explicitly to the metric definitions in the 'Evaluation Metrics' paragraph.
  4. [Section A.2] The description of the Blended trigger as 'an image of a certain popular character' is vague; please specify the character or object used and provide the exact blending operation and rate in addition to the value 0.2.
  5. [Algorithm 1, line 4] The algorithm says 'Sample P clean training samples' from D_clean but the poisoned set is later used to replace those samples; please clarify whether sampling is done without replacement and how this interacts with the definition of the poisoning rate gamma in Section 'Preliminaries'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the gradient-alignment objective is an optimization proxy, and ASR is an independent empirical transfer test.

full rationale

The paper's derivation chain is self-contained. The core objective (Eq. 3) minimizes the cosine distance between gradients of a frozen pretrained VLM on the target instance and on poisoned samples; Algorithm 1 turns this into concrete perturbations. ASR is then measured on held-out triggered inputs after LoRA fine-tuning. This is not circular: the training objective optimizes a gradient-alignment proxy, not the test-time output, so high ASR is an empirical result rather than a tautology. No fitted parameter is renamed as a prediction; the perturbations are optimized against Eq. 3 and evaluated with a separate behavioral metric. The skeptical concern that frozen-model gradient alignment may not transfer through LoRA fine-tuning is a legitimate correctness/robustness issue, but it is not circular reasoning. The self-citations to Liang et al. appear only in related work as context positioning VIBMA against prior VLM backdoor attacks; they are not load-bearing premises, uniqueness theorems, or ansatz justifications. Therefore no specified circular step is present; the paper's derivations and evaluations are independent of their inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The attack is empirical and does not present a mathematical derivation. Its success rests primarily on the transfer of frozen-model gradient alignment to actual fine-tuning, on the accuracy of auto-generated labels, and on attacker control of a fraction of the training corpus. The main chosen hyperparameters, epsilon, poison ratio, trigger size, and blending rate, materially affect the reported ASR and FSR, so they are listed as free parameters rather than constants.

free parameters (4)
  • perturbation budget epsilon = 8/255
    Chosen by the attacker; ablations from 4/255 to 16/255 show a trade-off between ASR and FSR, so the headline results depend on this choice.
  • poisoning ratio = 20%
    Fraction of poisoned fine-tuning samples; ablations show ASR drops to 80.49% at 10% and saturates near 30%, so the reported results depend on the 20% choice.
  • trigger size = 0.1% of screen (Hoverball), 2% (Hurdle bar)
    Ablations show trigger size affects the effectiveness and stealth trade-off; larger sizes improve ASR but hurt FSR.
  • Blended blending rate = 0.2
    Chosen opacity for the blended trigger; no ablation is reported for this value.
assumptions (5)
  • ad hoc to paper Gradient alignment on a frozen pretrained model transfers to LoRA fine-tuning on mixed clean and poisoned data.
    Algorithm 1 computes alignment on the frozen model, while the actual attack requires the poison to survive downstream fine-tuning. This transfer is asserted, not proven.
  • domain assumption The attacker can inject poisoned samples into the fine-tuning corpus without controlling the training process.
    Stated in the Threat Model under Attacker Capabilities.
  • domain assumption The automatically generated RICO prompts and action labels are accurate enough for reliable ASR and FSR measurement.
    Appendix A.3 and A.4 describe GPT-4-generated prompts and action inference by matching UI regions with prompt semantics.
  • domain assumption A small visual trigger overlaid on screenshots remains perceptible to the model while being imperceptible to users.
    Trigger Design and Figure 3 report PSNR and SSIM, but no human perceptual study is performed.
  • domain assumption The fine-tuning recipe used in the paper is representative of real mobile-agent adaptation pipelines.
    Appendix A.6 specifies LoRA, learning rate 2e-5, 10 epochs, batch size 4, and 6 GPUs; the generality of the attack to other adaptation setups is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents." pith.science (2026). https://pith.science/paper/CITOJAC4

@misc{pith2026250613205,
  author       = {Pith},
  title        = {Pith review of: Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CITOJAC4}},
  note         = {Machine review of arXiv:2506.13205}
}
read the original abstract

Mobile agents powered by vision-language models (VLMs) are increasingly adopted for tasks such as UI automation and camera-based assistance. These agents are typically fine-tuned using small-scale, user-collected data, making them susceptible to stealthy training-time threats. This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents. The attack injects malicious behaviors into the model by modifying only the visual input while preserving textual prompts and instructions, achieving stealth through the complete absence of textual anomalies. Once the agent is fine-tuned on this poisoned data, adding a predefined visual pattern (trigger) at inference time activates the attacker-specified behavior (backdoor). Our attack aligns the training gradients of poisoned samples with those of an attacker-specified target instance, effectively embedding backdoor-specific features into the poisoned data. To ensure the robustness and stealthiness of the attack, we design three trigger variants that better resemble real-world scenarios: static patches, dynamic motion patterns, and low-opacity blended content. Extensive experiments on six Android applications and three mobile-compatible VLMs demonstrate that our attack achieves high success rates (ASR up to 94.67%) while preserving clean-task behavior (FSR up to 95.85%). We further conduct ablation studies to understand how key design factors impact attack reliability and stealth. These findings is the first to reveal the security vulnerabilities of mobile agents and their susceptibility to backdoor injection, underscoring the need for robust defenses in mobile agent adaptation pipelines.

Figures

Figures reproduced from arXiv: 2506.13205 by the authors.

Figure 1
Figure 1. Overview of VIBMA. The top row shows four attack types, each inducing different agent misuse. The bottom row shows the training process (left), where imperceptible perturbations are optimized to generate poisoned images, which are then mixed with clean data to finetune the VLMs, and the test-time behavior (right), where a predefined trigger activates the backdoor and alters both the agent’s generated actions and rat… view at source ↗
Figure 2
Figure 2. Visualization of ASR and FSR across three trigger types (Hurdle, Hoverball, Blended) under different (a) applications [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Qualitative examples of triggered screenshots. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 24 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hassani, A.; and et al. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. arXiv preprint arXiv:2204.14198

    arXiv 2022

  4. [4]

    Chen, C.; Wang, B.; and Lin, Y. 2024. A Systematic Mapping Study of LLM Applications in Mobile Device Research. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, 163--174. Springer

    work page 2024

  5. [5]

    Cheng, P.; Wu, Z.; Du, W.; Zhao, H.; Lu, W.; and Liu, G. 2025. Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review. IEEE Transactions on Neural Networks and Learning Systems

    work page 2025

  6. [6]

    Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320--335

    work page 2022

  7. [7]

    G.; Zhang, Z.; Ma, S.; Zhang, J.; Fu, A.; Nepal, S.; and Kim, H

    Gao, Y.; Doan, B. G.; Zhang, Z.; Ma, S.; Zhang, J.; Fu, A.; Nepal, S.; and Kim, H. 2020. Backdoor attacks and countermeasures on deep learning: A comprehensive review. arXiv preprint arXiv:2007.10760

    arXiv 2020

  8. [8]

    H.; Huang, W

    Geiping, J.; Fowl, L. H.; Huang, W. R.; Czaja, W.; Taylor, G.; Moeller, M.; and Goldstein, T. 2021. Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching. In International Conference on Learning Representations

    work page 2021

Show all 43 references
  1. [9]

    Gu, T.; Liu, K.; Dolan-Gavitt, B.; and Garg, S. 2019. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7: 47230--47244

    2019

  2. [10]

    Huang, W.; Fei, F.; Savarese, S.; et al. 2022. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Proceedings of Robotics: Science and Systems (RSS)

    2022

  3. [11]

    R.; Geiping, J.; Fowl, L.; Taylor, G.; and Goldstein, T

    Huang, W. R.; Geiping, J.; Fowl, L.; Taylor, G.; and Goldstein, T. 2020. Metapoison: Practical general-purpose clean-label data poisoning. Advances in Neural Information Processing Systems, 33: 12080--12091

    2020

  4. [12]

    Jagielski, M.; Oprea, A.; Biggio, B.; Liu, C.; Nita-Rotaru, C.; and Li, B. 2018. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE symposium on security and privacy (SP), 19--35. IEEE

    2018

  5. [13]

    S.; Knox, W

    Lee, J.; Hahm, D.; Choi, J. S.; Knox, W. B.; and Lee, K. 2024 a . MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control. arXiv preprint arXiv:2410.17520

    2024

  6. [14]

    Lee, J.; Min, T.; An, M.; Hahm, D.; Lee, H.; Kim, C.; and Lee, K. 2024 b . Benchmarking Mobile Device Control Agents across Diverse Configurations. arXiv preprint arXiv:2404.16660

    2024 arXiv

  7. [15]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. C. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv preprint arXiv:2301.12597

    2023 arXiv

  8. [16]

    Li, Y.; Li, Y.; Wu, B.; Li, L.; He, R.; and Lyu, S. 2021. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF international conference on computer vision, 16463--16472

    2021

  9. [17]

    Li, Y.; Yang, Z.; Guo, Y.; and Chen, X. 2017. DroidBot: a lightweight UI-guided test input generator for Android. In Proceedings of the 39th International Conference on Software Engineering Companion, ICSE-C '17, 23–26. IEEE Press. ISBN 9781538615898

    2017

  10. [18]

    Liang, J.; Liang, S.; Liu, A.; and Cao, X. 2025. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. International Journal of Computer Vision, 1--20

    2025

  11. [19]

    Liang, S.; Liang, J.; Pang, T.; Du, C.; Liu, A.; Chang, E.-C.; and Cao, X. 2024. Revisiting backdoor attacks against large vision-language models. arXiv preprint arXiv:2406.18844

    2024 arXiv

  12. [20]

    Liang, S.; Pang, T.; Cao, X.; and Liu, A. 2023. Mimic and Fool: A Simple yet Effective Fine-tuning Approach for Reducing Object Hallucination in Vision-Language Models. arXiv preprint arXiv:2310.02234

    2023 arXiv

  13. [21]

    Liu, H.; Zhang, C.; Du, Y.; Lin, Y.; Li, J.; Wang, Z.; Hu, Z.; Wang, J.; and Gao, J. 2023. Language-vision alignment for instruction-following with llava. arXiv preprint arXiv:2304.08485

    2023 arXiv

  14. [22]

    F.; Craft, M.; Situ, J.; Yumer, E.; Mech, R.; and Kumar, R

    Liu, T. F.; Craft, M.; Situ, J.; Yumer, E.; Mech, R.; and Kumar, R. 2018 a . Learning Design Semantics for Mobile Apps. In The 31st Annual ACM Symposium on User Interface Software and Technology, UIST '18, 569--579

    2018

  15. [23]

    Liu, Y.; Ma, S.; Aafer, Y.; Lee, W.-C.; Zhai, J.; Wang, W.; and Zhang, X. 2018 b . Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc

    2018

  16. [24]

    Liu, Y.; et al. 2024. AgentScope: A Flexible yet Robust Multi-Agent Platform. arXiv preprint arXiv:2402.14034

    2024 arXiv

  17. [25]

    Nguyen, A.; and Tran, A. 2021. Wanet--imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369

    2021 arXiv

  18. [26]

    Qin, T.; Wang, X.; Zhao, J.; Ye, K.; Xu, C.-Z.; and Gao, X. 2025. On the Adversarial Robustness of Visual-Language Chat Models. In Zhang, Z. M.; Ricci, E.; Yan, Y.; Nie, L.; Oria, V.; and Ballan, L., eds., Proceedings of the 2025 International Conference on Multimedia Retrieva...

    2025

  19. [27]

    Rawles, C.; Li, A.; Rodriguez, D.; Riva, O.; and Lillicrap, T. 2023. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36: 59708--59728

    2023

  20. [28]

    Saha, A.; Subramanya, A.; and Pirsiavash, H. 2020. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 11957--11965

    2020

  21. [29]

    Tian, Z.; Cui, L.; Liang, J.; and Yu, S. 2022. A comprehensive survey on poisoning attacks and countermeasures in machine learning. ACM Computing Surveys, 55(8): 1--35

    2022

  22. [30]

    Turner, A.; Tsipras, D.; and Madry, A. 2018. Clean-label backdoor attacks

    2018

  23. [31]

    Wang, J.; Xu, H.; Ye, J.; Yan, M.; Shen, W.; Zhang, J.; Huang, F.; and Sang, J. 2024 a . Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158

    2024 arXiv

  24. [32]

    Wang, Y.; Xue, D.; Zhang, S.; and Qian, S. 2024 b . BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9811--9827

    2024

  25. [33]

    Wang, Z.; Xu, H.; Wang, J.; Zhang, X.; Yan, M.; Zhang, J.; Huang, F.; and Ji, H. 2025. Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks. arXiv preprint arXiv:2501.11733

    2025 arXiv

  26. [34]

    Wei, J.; et al. 2023. Jailbroken: How Does LLM Safety Training Fail? arXiv preprint arXiv:2307.02483

    2023 arXiv

  27. [36]

    Xu, Y.; Yao, J.; Shu, M.; Sun, Y.; Wu, Z.; Yu, N.; Goldstein, T.; and Huang, F. 2024. Shadowcast: Stealthy data poisoning attacks against vision-language models. arXiv preprint arXiv:2402.06659

    2024 arXiv

  28. [37]

    Yang, W.; Bi, X.; Lin, Y.; Chen, S.; Zhou, J.; and Sun, X. 2024 a . Watch out for your agents! investigating backdoor threats to llm-based agents. Advances in Neural Information Processing Systems, 37: 100938--100964

    2024

  29. [38]

    Yang, Y.; Yang, X.; Li, S.; Lin, C.; Zhao, Z.; Shen, C.; and Zhang, T. 2024 b . Security matrix for multimodal agents on mobile devices: A systematic and proof of concept study. arXiv preprint arXiv:2407.09295

    2024 arXiv

  30. [39]

    Yao, S.; Chen, H.; Yang, J.; and Narasimhan, K. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)

    2022

  31. [40]

    M.; and Jia, R

    Zeng, Y.; Park, W.; Mao, Z. M.; and Jia, R. 2021. Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 16473--16481

    2021

  32. [41]

    Zhang, C.; Yang, Z.; Liu, J.; Li, Y.; Han, Y.; Chen, X.; Huang, Z.; Fu, B.; and Yu, G. 2025. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1--20

    2025

  33. [42]

    Zhao, S.; Ma, X.; Zheng, X.; Bailey, J.; Chen, J.; and Jiang, Y.-G. 2020. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14443--14452

    2020

  34. [43]

    Zhou, F.; Liu, Y.; Cheng, Z.; et al. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv preprint arXiv:2404.07972

    2024 arXiv

  35. [44]

    C.; Liu, Y.; Liu, S.; Wang, Y.; Xu, C.; et al

    Zhu, D.; Yang, J. C.; Liu, Y.; Liu, S.; Wang, Y.; Xu, C.; et al. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.