Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

A backdoor can force GUI agents to reason for far longer without changing their final clicks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 12:41 UTC pith:JYESUFPP

load-bearing objection First efficiency backdoor on VLM GUI agents with a clean two-stage RBI method and multi-domain numbers; Web accuracy drop is real but does not erase the contribution. the 3 major comments →

arxiv 2603.08316 v3 pith:JYESUFPP submitted 2026-03-09 cs.CR cs.CLcs.CV

SlowBA: An efficiency backdoor attack towards VLM-based GUI agents

Junxian Li , Tu Lan , Haozhen Tan , Yan Meng , Haojin Zhu This is my paper
classification cs.CR cs.CLcs.CV
keywords backdoor attackefficiency attackVLM-based GUI agentsresponse latencyreward-level injectionpop-up triggersreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

VLM-based GUI agents are sold on both accuracy and low latency, but security research has almost only attacked the first. This paper claims a second attack surface: an efficiency backdoor that makes the agent generate extremely long reasoning chains when a realistic pop-up trigger appears on the screen. The method, called SlowBA, uses a two-stage training recipe—first teaching a long-response format by supervised fine-tuning on triggered screenshots, then using reinforcement learning with a length-based reward that fires only on triggered inputs. On web, desktop, and Android datasets the attack multiplies response length and latency while clean accuracy stays close to the original model and triggered accuracy stays close to clean accuracy. Pop-up triggers look like ordinary notifications, and the effect survives low poisoning ratios and several common defenses. The authors argue that defenses must therefore protect responsiveness as well as action correctness.

Core claim

SlowBA injects a backdoor into VLM-based GUI agents so that, under realistic pop-up triggers, the model produces much longer reasoning chains and therefore much higher latency and energy use, while still emitting largely correct final actions on both clean and triggered inputs.

What carries the argument

Two-stage reward-level backdoor injection (RBI): Stage I supervised fine-tuning aligns the model to a long-response format on triggered samples; Stage II GRPO-style reinforcement learning applies a length reward only to triggered inputs (positive for long triggered replies, zero or negative for long clean replies), so length amplification is gated by the trigger.

Load-bearing premise

Response length is a reliable proxy for latency, based on a strong measured correlation on the authors' agent samples; if that link weakens under other models or decoding settings, length rewards will not deliver the intended slowdown.

What would settle it

On a held-out GUI agent and decoding configuration where response length and wall-clock latency are only weakly correlated, check whether the same RBI training still produces large I-latency under the pop-up trigger while preserving action accuracy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SlowBA, a backdoor attack on VLM-based GUI agents that targets response efficiency (latency/energy) rather than action correctness. The attack reformulates high latency as response-length maximization (supported by a measured Pearson correlation of r=0.8059 on GUI-R1 samples) and injects the backdoor via a two-stage reward-level backdoor injection (RBI) pipeline: Stage I SFT aligns a long-response format on triggered samples, and Stage II GRPO-style RL applies a trigger-aware length reward (Eq. 3) that boosts length only on triggered inputs while penalizing long clean responses. Triggers are adaptive rendered pop-up notifications (domain-aware for web; update/restart for desktop/app). Experiments on GUI-R1-3B (plus a 7B scale-up) across Web, Desktop, and Android report large relative increases in length/latency/energy (e.g., I-length 358.52%, I-latency 66.92% on Web) at 0.1 poisoning ratio, with clean Acc near the original model; the attack is ablated by stage, tested under several defenses, module-wise injection, human stealthiness rating of the pop-ups, and a small real-world 12306 ticket-booking demo. Code is released.

Significance. Efficiency backdoors on VLM-based GUI agents are largely unexplored; the paper correctly identifies a realistic threat model (open model hubs, visual-only triggers, SFT+RL finetuning) and supplies a concrete, reproducible attack with multi-domain evaluation, stage ablations, defense suite, human trigger study, and a limited real-world demo. If the dual claim (large latency inflation + usable accuracy under trigger) holds more broadly, the work usefully expands the GUI-agent threat surface beyond action-level backdoors and motivates efficiency-aware defenses. Strengths that should be credited include the explicit length–latency correlation check, the disentangled RBI design, released code, and the realistic pop-up trigger pipeline rather than pure-color/Gaussian patches.

major comments (3)
  1. [Table 1, Eq. (1), §3.1, §5.2] Table 1 (Web row) and the dual objective in Eq. (1)/§3.1: the strongest claim that SlowBA “largely preserv[es] task accuracy” and that triggered Acc stays close to clean Acc is only partially supported. On Web—the domain with the largest efficiency gains—triggered Acc falls to 49.3 from the clean model’s 60.5 (and clean Acc is 63.1), an absolute drop of ~11 points. Desktop and Android are closer, but the primary evaluation domain violates the stated stealthiness constraint. The abstract and §5.2 should qualify the claim by domain, report absolute Acc deltas, and discuss whether the Web drop is acceptable under the threat model or can be reduced (e.g., by reintroducing a light accuracy term in Eq. 3).
  2. [§5.1, §5.7] §5.1 and §5.7: evaluation is effectively confined to one agent family (GUI-R1 3B/7B, Qwen2.5-VL backbone). Module-wise injection and 7B scale-up are useful but do not establish transfer to other VLM GUI agents (different RL recipes, grounding heads, or SFT-only pipelines). A load-bearing claim of a general efficiency vulnerability for “VLM-based GUI agents” needs at least one additional independent agent or a clear limitation statement that results are GUI-R1-specific.
  3. [§4.1, Fig. 2, Eq. (3)] §4.1 / Fig. 2: the attack optimizes length as a proxy for latency after measuring r=0.8059 on (clean) GUI-R1 samples. This is the central modeling assumption. The paper should report the same correlation under the backdoored model and under the decoding settings used at inference, and discuss failure modes if the correlation weakens (e.g., early EOS, different temperature, or agents that already emit long CoT). Without that, the mapping from the reward in Eq. (3) to the stated latency objective remains incompletely validated.
minor comments (5)
  1. [§5.1] Baselines (Gaussian, JPEG, Verbose Image, adapted VisualTrap) are reasonable given the lack of prior efficiency backdoors, but §5.1 should more explicitly state that none is a true length-maximizing backdoor, so relative ranking is mainly illustrative.
  2. [Fig. 5, Fig. 6] Fig. 5 and Fig. 6 captions/axes are hard to read in the manuscript text; ensure token-length distributions and Grad-CAM pairs are labeled with absolute means and the query used.
  3. [Eq. (3), Appendix B] Appendix reward ablation (α, β) is useful; consider moving a short summary into the main text near Eq. (3) so readers see sensitivity without opening the appendix.
  4. [§4] Minor wording: “responsetexts”, “high-latencygoal”, and similar concatenated tokens appear in §4; clean for camera-ready.
  5. [§5.7] Real-world 12306 experiment (§5.7) is suggestive but single-run; report variance over multiple bookings or seeds if available.

Circularity Check

0 steps flagged

No circularity: empirical backdoor construction whose length-maximizing reward is justified by an independent correlation measurement and then validated by separate latency/energy metrics.

full rationale

SlowBA is an empirical security paper. The attack objective (Eq. 1) is to maximize Latency under triggers subject to accuracy constraints; because direct latency optimization is awkward, the authors measure a Pearson correlation r = 0.8059 between length and latency on clean GUI-R1 samples (Fig. 2, §4.1) and therefore substitute a length-based reward (Eq. 3) that is applied only to triggered inputs. That substitution is an engineering proxy, not a definitional identity: length and latency remain independently measured quantities (I-length, I-latency, I-energy) reported side-by-side in Tables 1–5. Stage-I SFT merely teaches a long-response format; Stage-II GRPO then selectively amplifies length under the trigger. Neither stage imports a uniqueness theorem, renames a known result, nor treats a fitted constant as a prediction. Self-citations are ordinary related-work references (GUI-R1, GRPO/DeepSeek-R1, VisualTrap, etc.) and are not load-bearing for the central claim. The experimental results (accuracy preservation, defense robustness, real-world ticket-booking latency) are external measurements, not tautological consequences of the reward definition. Consequently the derivation chain contains no circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard backdoor threat models, the measured length–latency correlation, GRPO-style RL, and a handful of hand-chosen reward and poisoning hyperparameters. No new physical entities; the invented pieces are the attack procedure and trigger construction.

free parameters (4)
  • reward alpha = 2
    Scales positive length reward on triggered inputs; set to 2 empirically (Eq. 3, Appendix B).
  • reward beta = 1/8
    Length threshold fraction of l_max below which clean long answers get zero reward; set to 1/8 empirically.
  • poisoning ratio = 0.1
    Fraction of triggered samples in SFT and RL stages; fixed at 0.1 following prior work.
  • l_max = 8192
    Maximum sequence length used to normalize the length reward; set to 8192.
axioms (4)
  • domain assumption Response latency of the target VLM GUI agent is strongly positively correlated with generated token length, so maximizing length under trigger is a valid surrogate for maximizing latency.
    Stated and measured (r=0.8059) in §4.1 / Fig. 2; load-bearing for the optimization reformulation.
  • domain assumption Attacker can fine-tune a pretrained GUI agent with both SFT and RL (GRPO) on a small poisoned visual set while freezing or not freezing vision modules.
    Threat model §3.2; standard for training-time backdoors on open models.
  • domain assumption GRPO group-relative advantages with KL regularization to the Stage-I reference policy stably increase length only on triggered inputs without collapsing action format.
    Used in §4.5 and Appendix C; inherited from DeepSeek-R1 / GUI-R1 training practice.
  • ad hoc to paper Rendered pop-up notifications (domain-aware for web, update/restart for desktop/app) are visually normal enough that users and simple filters do not remove them.
    Trigger design §4.3 and human evaluation §5.7; required for Availability and stealthiness claims.
invented entities (2)
  • RBI (two-stage reward-level backdoor injection) no independent evidence
    purpose: Disentangle long-response format learning (Stage I SFT) from trigger-conditional length activation (Stage II RL reward).
    Core methodological contribution; no independent existence outside this attack pipeline.
  • Adaptive pop-up trigger pipeline (Qwen3-VL domain extraction + Playwright rendering) no independent evidence
    purpose: Produce stealthy, environment-plausible visual triggers for web/desktop/app screenshots.
    Introduced in §4.3 to satisfy Availability; realism supported only by the paper’s own human study.

pith-pipeline@v1.1.0-grok45 · 22944 in / 3001 out tokens · 30485 ms · 2026-07-15T12:41:39.539774+00:00 · methodology

0 comments
read the original abstract

Modern vision-language-model (VLM) based graphical user interface (GUI) agents are expected not only to execute actions accurately but also to respond to user instructions with low latency. While existing research on GUI-agent security mainly focuses on manipulating action correctness, the security risks related to response efficiency remain largely unexplored. In this paper, we introduce SlowBA, a novel backdoor attack that targets the responsiveness of VLM-based GUI agents. The key idea is to manipulate response latency by inducing excessively long reasoning chains under specific trigger patterns. To achieve this, we propose a two-stage reward-level backdoor injection (RBI) strategy that first aligns the long-response format and then learns trigger-aware activation through reinforcement learning. In addition, we design realistic pop-up windows as triggers that naturally appear in GUI environments, improving the stealthiness of the attack. Extensive experiments across multiple datasets and baselines demonstrate that SlowBA can significantly increase response length and latency while largely preserving task accuracy. The attack remains effective even with a small poisoning ratio and under several defense settings. These findings reveal a previously overlooked security vulnerability in GUI agents and highlight the need for defenses that consider both action correctness and response efficiency. Code can be found in https://github.com/tu-tuing/SlowBA.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents

    cs.CL 2026-04 unverdicted novelty 7.0

    OS-SPEAR is a new evaluation toolkit that tests 22 OS agents and identifies trade-offs between efficiency and safety or robustness.

  2. Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability

    cs.CL 2026-05 unverdicted novelty 4.0

    The paper develops a unified framework that organizes computer-use agent reliability around perception-decision-execution layers and creation-deployment-operation-maintenance stages to map security and alignment inter...

Reference graph

Works this paper leans on

74 extracted references · 18 linked inside Pith · cited by 2 Pith papers

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023) 2, 4

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 2, 4

    Pith/arXiv arXiv 2023

  2. [2]

    An, R., Yang, S., Guo, Z., Dai, W., Shen, Z., Li, H., Zhang, R., Wei, X., Li, G., Wu,W.,etal.:Genius:Generativefluidintelligenceevaluationsuite.arXivpreprint arXiv:2602.11144 (2026) 4

    arXiv 2026

  3. [3]

    arXiv preprint arXiv:2411.11706 (2024) 4

    An, R., Yang, S., Lu, M., Zhang, R., Zeng, K., Luo, Y., Cao, J., Liang, H., Chen, Y., She, Q., et al.: Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706 (2024) 4

    arXiv 2024

  4. [4]

    arXiv preprint arXiv:2505.14671 (2025) 4

    An, R., Yang, S., Zhang, R., Shen, Z., Lu, M., Dai, G., Liang, H., Guo, Z., Yan, S., Luo, Y., et al.: Unictokens: Boosting personalized understanding and generation via unified concept tokens. arXiv preprint arXiv:2505.14671 (2025) 4

    arXiv 2025

  5. [5]

    arXiv preprint arXiv:2509.23661 (2025) 4

    An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Zhu, D., et al.: Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661 (2025) 4

    Pith/arXiv arXiv 2025

  6. [6]

    arXiv preprint arXiv:2503.10200 (2025) 4

    Chen, B., Yue, Z., Chen, S., Wang, Z., Liu, Y., Li, P., Wang, Y.: Lvagent: Long video understanding by multi-round dynamical collaboration of mllm agents. arXiv preprint arXiv:2503.10200 (2025) 4

    arXiv 2025

  7. [7]

    arXiv preprint arXiv:2512.00756 (2025) 2

    Chen, R., Li, Q., Feng, X., Yang, X., Zhong, W., Gu, Y., Zhou, Z., Qin, B.: Mpr- gui: Benchmarking and enhancing multilingual perception and reasoning in gui agents. arXiv preprint arXiv:2512.00756 (2025) 2

    Pith/arXiv arXiv 2025

  8. [8]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, S., Chen, H., Haque, M., Liu, C., Yang, W.: The dark side of dynamic routing neural networks: Towards efficiency backdoor injection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24585–24594 (2023) 4

    2023

  9. [9]

    arXiv preprint arXiv:2509.11173 (2025) 5 16 Li et al

    Chen, S., Peng, J., He, Y., Yang, J., Ray, B.: Your compiler is backdooring your model: Understanding and exploiting compilation inconsistency vulnerabilities in deep learning compilers. arXiv preprint arXiv:2509.11173 (2025) 5 16 Li et al

    arXiv 2025

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, S., Song, Z., Haque, M., Liu, C., Yang, W.: Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15365–15374 (2022) 10

    2022

  11. [11]

    In: Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers)

    Chen, W., Cui, J., Hu, J., Qin, Y., Fang, J., Zhao, Y., Wang, C., Liu, J., Chen, G., Huo, Y., et al.: Guicourse: From general vision language model to versatile gui agent. In: Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers). pp. 21936–21959 (2025) 8, 10

    2025

  12. [12]

    arXiv preprint arXiv:2505.14418 (2025) 4

    Cheng, P., Hu, H., Wu, Z., Wu, Z., Ju, T., Zhang, Z., Liu, G.: Hidden ghost hand: Unveiling backdoor vulnerabilities in mllm-powered mobile gui agents. arXiv preprint arXiv:2505.14418 (2025) 4

    arXiv 2025

  13. [13]

    IEEE Transactions on Neural Networks and Learning Systems (2025) 4

    Cheng, P., Wu, Z., Du, W., Zhao, H., Lu, W., Liu, G.: Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review. IEEE Transactions on Neural Networks and Learning Systems (2025) 4

    2025

  14. [14]

    In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    Das, N., Shanbhogue, M., Chen, S.T., Hohman, F., Li, S., Chen, L., Kounavis, M.E., Chau, D.H.: Shield: Fast, practical defense and vaccination for deep learning using jpeg compression. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 196–204 (2018) 12

    2018

  15. [15]

    In: Proceedings of the 2025 International Conference on Multimedia Retrieval

    Ding, T., He, L., Ma, W., Zhou, X.: Musellm: Sdf generation and understanding via multi-scale tokenization with position-aware guidance. In: Proceedings of the 2025 International Conference on Multimedia Retrieval. p. 192–201. ICMR ’25, Association for Computing Machinery, New York, NY, USA (2025).https://doi. org/10.1145/3731715.3733405,https://doi.org/1...

    doi:10.1145/3731715.3733405 2025

  16. [16]

    Falotico,R.,Quatto,P.:Fleiss’kappastatisticwithoutparadoxes.Quality&Quan- tity49(2), 463–470 (2015) 13

    2015

  17. [17]

    In: The Twelfth In- ternational Conference on Learning Representations (2024) 10

    Gao, K., Bai, Y., Gu, J., Xia, S.T., Torr, P., Li, Z., Liu, W.: Inducing high energy- latency of large vision-language models with verbose images. In: The Twelfth In- ternational Conference on Learning Representations (2024) 10

    2024

  18. [18]

    In: The Thirteenth International Conference on Learning Representations (2025) 2

    Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu, Y., Sun, H., Su, Y.: Navigating the digital world as humans do: Universal visual grounding for gui agents. In: The Thirteenth International Conference on Learning Representations (2025) 2

    2025

  19. [19]

    arXiv preprint arXiv:2312.06677 (2023) 1, 4

    Guan, Y., Wang, D., Chu, Z., Wang, S., Ni, F., Song, R., Li, L., Gu, J., Zhuang, C.: Intelligent virtual assistants with llm-based process automation. arXiv preprint arXiv:2312.06677 (2023) 1, 4

    Pith/arXiv arXiv 2023

  20. [20]

    Nature645(8081), 633–638 (2025) 3, 6, 9, 21

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature645(8081), 633–638 (2025) 3, 6, 9, 21

    2025

  21. [21]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al.: Cogagent: A visual language model for gui agents. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14281–14290 (2024) 2

    2024

  22. [22]

    In: International Conference on Learning Representations (2022) 10

    Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022) 10

    2022

  23. [23]

    In: European Conference on Computer Vision

    Kapoor, R., Butala, Y.P., Russak, M., Koh, J.Y., Kamble, K., AlShikh, W., Salakhutdinov, R.: Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In: European Conference on Computer Vision. pp. 161–178. Springer (2024) 8, 10 SlowBA: An efficiency backdoor attack towards VLM-based GUI agents 17

    2024

  24. [24]

    arXiv preprint arXiv:2508.09456 (2025) 2, 4

    Li, J., Xu, B., Chen, S., Li, J., Lei, J., Zhao, H., Zhang, D.: Iag: Input-aware backdoor attack on vlm-based visual grounding. arXiv preprint arXiv:2508.09456 (2025) 2, 4

    Pith/arXiv arXiv 2025

  25. [25]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Li, J., Zhang, D., Wang, X., Hao, Z., Lei, J., Tan, Q., Zhou, C., Liu, W., Yang, Y., Xiong, X., et al.: Chemvlm: Exploring the power of multimodal large language models in chemistry area. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 415–423 (2025) 4

    2025

  26. [26]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Li, K., Meng, Z., Lin, H., Luo, Z., Tian, Y., Ma, J., Huang, Z., Chua, T.S.: Screenspot-pro: Gui grounding for professional high-resolution computer use. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 8778– 8786 (2025) 10

    2025

  27. [27]

    arXiv preprint arXiv:2511.05923 (2025) 4

    Li, Q., Ye, Z., Feng, X., Zhong, W., Ma, W., Feng, X.: Causal tracing of object representations in large vision language models: Mechanistic interpretability and hallucination mitigation. arXiv preprint arXiv:2511.05923 (2025) 4

    arXiv 2025

  28. [28]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Li, S., Hu, Y., Ning, X., Liu, X., Hong, K., Jia, X., Li, X., Yan, Y., Ran, P., Dai, G., et al.: Mbq: Modality-balanced quantization for large vision-language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4167–4177 (2025) 12

    2025

  29. [29]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, S., Ma, W., Guo, J., Xu, S., Li, B., Zhang, X.: Unionformer: Unified-learning transformer with multi-view representation for image manipulation detection and localization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12523–12533 (2024) 4

    2024

  30. [30]

    Li,S.,Xing,Z.,Wang,H.,Hao,P.,Li,X.,Liu,Z.,Zhu,L.:Towardmedicaldeepfake detection:Acomprehensivedatasetandnovelmethod.In:InternationalConference on Medical Image Computing and Computer-Assisted Intervention. pp. 626–637. Springer (2025) 4

    2025

  31. [31]

    arXiv preprint arXiv:2406.036795(7) (2024) 10

    Li, W., Bishop, W., Li, A., Rawles, C., Campbell-Ajala, F., Tyamagundlu, D., Riva, O.: On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.036795(7) (2024) 10

    Pith/arXiv arXiv 2024

  32. [32]

    IEEE Transactions on Neural Networks and Learning Systems36, 19525–19545 (2024) 4

    Liu, D., Yang, M., Qu, X., Zhou, P., Hu, W., Cheng, Y.: A survey of attacks on large vision–language models: Resources, advances, and future trends. IEEE Transactions on Neural Networks and Learning Systems36, 19525–19545 (2024) 4

    2024

  33. [33]

    Advances in neural information processing systems36, 34892–34916 (2023) 2

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023) 2

    2023

  34. [34]

    In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Liu, Z., Liu, Q., Liu, T., Xu, N., Lin, X., Wang, Y., Wen, W.: Feature distillation: Dnn-oriented jpeg compression against adversarial examples. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 860–868. IEEE (2019) 10

    2019

  35. [35]

    arXiv preprint arXiv:2504.10458 (2025) 2, 5, 6, 9, 10, 22

    Luo, R., Wang, L., He, W., Chen, L., Li, J., Xia, X.: Gui-r1: A generalist r1- style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458 (2025) 2, 5, 6, 9, 10, 22

    Pith/arXiv arXiv 2025

  36. [36]

    In: European Conference on Computer Vision

    Lyu, W., Pang, L., Ma, T., Ling, H., Chen, C.: Trojvlm: Backdoor attack against vision language models. In: European Conference on Computer Vision. pp. 467–

  37. [37]

    Springer (2024) 2, 5, 7

    2024

  38. [38]

    In: The Thir- teenth International Conference on Learning Representations (2025) 12

    Lyu, W., Yao, J., Gupta, S., Pang, L., Sun, T., Yi, L., Hu, L., Ling, H., Chen, C.: Backdooring vision-language models with out-of-distribution data. In: The Thir- teenth International Conference on Learning Representations (2025) 12

    2025

  39. [39]

    In: 30th Annual Network 18 Li et al

    Ma, W., Wang, D., Sun, R., Xue, M., Wen, S., Xiang, Y.: The "beatrix" resur- rections: Robust backdoor detection via gram matrices. In: 30th Annual Network 18 Li et al. and Distributed System Security Symposium, NDSS 2023, San Diego, California, USA, February 27 - March 3, 2023. The Internet Society (2023) 12

    2023

  40. [40]

    Ma, W., Sun, S., Yu, T., Wang, R., Chua, T.S., Bian, J.: Thinking with blueprints: Assisting vision-language models in spatial reasoning via structured object repre- sentation (2026),https://arxiv.org/abs/2601.019844

    arXiv 2026

  41. [41]

    arXiv preprint arXiv:2402.11941 (2024) 4

    Ma, X., Zhang, Z., Zhao, H.: Coco-agent: A comprehensive cognitive mllm agent for smartphone gui automation. arXiv preprint arXiv:2402.11941 (2024) 4

    Pith/arXiv arXiv 2024

  42. [42]

    News, Y.: As spring festival travel tickets go on sale, many train routes departing from ningbo are sold out within seconds.https://m.thepaper.cn/newsDetail_ forward_32414831(2026), accessed: 2026-03-05 15

    2026

  43. [43]

    In: Trustworthy Multi-modal Foundation Models and AI Agents (TiFA) (2025) 2, 5

    Ni, Z., Ye, R., Wei, Y., Xiang, Z., Wang, Y., Chen, S.: Physical backdoor attack can jeopardize driving with vision-large-language models. In: Trustworthy Multi-modal Foundation Models and AI Agents (TiFA) (2025) 2, 5

    2025

  44. [44]

    In: Proceedings of the IEEE international conference on computer vision

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017) 12

    2017

  45. [45]

    In: Proceedings of the Twentieth European Conference on Computer Systems

    Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., Wu, C.: Hybridflow: A flexible and efficient rlhf framework. In: Proceedings of the Twentieth European Conference on Computer Systems. pp. 1279–1297 (2025) 10

    2025

  46. [46]

    arXiv preprint arXiv:2507.05720 (2025) 2, 9

    Shi, Y., Yu, W., Li, Z., Wang, Y., Zhang, H., Liu, N., Mi, H., Yu, D.: Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment. arXiv preprint arXiv:2507.05720 (2025) 2, 9

    Pith/arXiv arXiv 2025

  47. [47]

    Team, M.: Playwright for python.https://playwright.dev/python/, accessed: 2026-01-31 7

    2026

  48. [48]

    Team, Q.: Qwen2.5-vl (January 2025),https://qwenlm.github.io/blog/qwen2. 5-vl/9

    2025

  49. [49]

    Team, T.M.: Modelscope: bring the notion of model-as-a-service to life.https: //github.com/modelscope/modelscope(2023) 2

    2023

  50. [50]

    Advances in neural information processing systems31(2018) 12

    Tran, B., Li, J., Madry, A.: Spectral signatures in backdoor attacks. Advances in neural information processing systems31(2018) 12

    2018

  51. [51]

    IEEE Transactions on Information Theory60(7), 3797–3820 (2014) 9

    Van Erven, T., Harremos, P.: Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory60(7), 3797–3820 (2014) 9

    2014

  52. [52]

    arXiv preprint arXiv:2409.12191 (2024) 4

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024) 4

    Pith/arXiv arXiv 2024

  53. [53]

    Advances in Neural Information Processing Systems37, 121475–121499 (2024) 4

    Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al.: Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems37, 121475–121499 (2024) 4

    2024

  54. [54]

    arXiv preprint arXiv:2411.11683 (2024) 2

    Wang, X., Pan, H., Zhang, H., Li, M., Hu, S., Zhou, Z., Xue, L., Liu, A., Jiang, Y., Zhang, L.Y., et al.: Trojanrobot: Physical-world backdoor attacks against vlm- based robotic manipulation. arXiv preprint arXiv:2411.11683 (2024) 2

    arXiv 2024

  55. [55]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Wang, X., Yao, T., Chen, S., Wang, R., Ye, L., Gao, K., Huang, Y., Yao, Y.: Vlminferslow: Evaluating the efficiency robustness of large vision-language models as a service. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 16035–16050 (2025) 10

    2025

  56. [56]

    In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. pp. 38–45 (2020) 2 SlowBA: An efficiency backdoor attack t...

    2020

  57. [57]

    Wu, Z., Cheng, P., Wu, Z., Dong, L., Zhang, Z.: Gem: Gaussian embedding mod- eling for out-of-distribution detection in gui agents (2025),https://arxiv.org/ abs/2505.128422

    arXiv 2025

  58. [58]

    Wu, Z., Huang, H., Lou, X., Qu, X., Cheng, P., Wu, Z., Liu, W., Zhang, W., Wang, J., Wang, Z., Zhang, Z.: Verios: Query-driven proactive human-agent-gui interaction for trustworthy os agents (2025),https://arxiv.org/abs/2509.07553 4

    Pith/arXiv arXiv 2025

  59. [59]

    arXiv preprint arXiv:1704.01155 (2017) 10, 12

    Xu, W., Evans, D., Qi, Y.: Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155 (2017) 10, 12

    Pith/arXiv arXiv 2017

  60. [60]

    arXiv preprint arXiv:2505.09388 (2025) 7, 8

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) 7, 8

    Pith/arXiv arXiv 2025

  61. [61]

    arXiv preprint arXiv:2505.23762 (2025) 4, 5

    Yang, C., Su, S., Liu, S., Dong, X., Yu, Y., Su, W., Wang, X., Liu, Z., Zhu, J., Li, H., et al.: Zerogui: Automating online gui learning at zero human cost. arXiv preprint arXiv:2505.23762 (2025) 4, 5

    Pith/arXiv arXiv 2025

  62. [62]

    arXiv preprint arXiv:2508.15144 (2025) 1, 4

    Ye, J., Zhang, X., Xu, H., Liu, H., Wang, J., Zhu, Z., Zheng, Z., Gao, F., Cao, J., Lu, Z., et al.: Mobile-agent-v3: Fundamental agents for gui automation. arXiv preprint arXiv:2508.15144 (2025) 1, 4

    Pith/arXiv arXiv 2025

  63. [63]

    arXiv preprint arXiv:2507.06899 (2025) 2, 4, 7, 10

    Ye, Z., Zhang, Y., Shi, W., You, X., Feng, F., Chua, T.S.: Visualtrap: A stealthy backdoor attack on gui agents via visual grounding manipulation. arXiv preprint arXiv:2507.06899 (2025) 2, 4, 7, 10

    arXiv 2025

  64. [64]

    arXiv preprint arXiv:2510.15863 (2025) 4

    Yu, S., Li, G., Shi, W., Qi, P.: Polyskill: Learning generalizable skills through polymorphic abstraction. arXiv preprint arXiv:2510.15863 (2025) 4

    arXiv 2025

  65. [65]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Yuan, Z., Shi, J., Zhou, P., Gong, N.Z., Sun, L.: Badtoken: Token-level backdoor attacks to multi-modal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 29927– 29936 (June 2025) 4

    2025

  66. [66]

    arXiv preprint arXiv:2510.27623 (2025) 4

    Zhan, Q., Ha, H., Yang, R., Xu, S., Chen, H., Gui, L.Y., Wang, Y.X., Zhang, H., Ji, H., Kang, D.: Beat: Visual backdoor attacks on vlm-based embodied agents via contrastive trigger learning. arXiv preprint arXiv:2510.27623 (2025) 4

    arXiv 2025

  67. [67]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, D., Lei, J., Li, J., Wang, X., Liu, Y., Yang, Z., Li, J., Wang, W., Yang, S., Wu, J., et al.: Critic-v: Vlm critics help catch vlm errors in multimodal reasoning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9050–9061 (2025) 4

    2025

  68. [68]

    In: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations)

    Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z.: Llamafactory: Unified efficient fine-tuning of 100+ language models. In: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations). pp. 400–410 (2024) 10

    2024

  69. [69]

    arXiv preprint arXiv:2506.07214 (2025) 2

    Zhong, Z., Sun, Z., Liu, Y., He, X., Tao, G.: Backdoor attack on vision language models with stealthy semantic manipulation. arXiv preprint arXiv:2506.07214 (2025) 2

    Pith/arXiv arXiv 2025

  70. [70]

    arXiv preprint arXiv:2504.21277 (2025) 9

    Zhou, G., Qiu, P., Chen, C., Wang, J., Yang, Z., Xu, J., Qiu, M.: Reinforced mllm: A survey on rl-based reasoning in multimodal large language models. arXiv preprint arXiv:2504.21277 (2025) 9

    Pith/arXiv arXiv 2025

  71. [71]

    Zhou, X., Tie, G., Zhang, G., Wang, H., Zhou, P., Sun, L.: Badvla: Towards back- door attacks on vision-language-action models via objective-decoupled optimiza- tion. arXiv preprint arXiv:2505.16640 (2025) 5 Technical Appendix No Author Given No Institute Given A Prompts for Data Generation in Stage I You are a rewriter and you are given a screenshot of a...

    Pith/arXiv arXiv 2025

  72. [72]

    Specifically analyze every object on the screen,

  73. [73]

    Create a logic cycle and repeat it several times,

  74. [74]

    type”, “select

    Add some irrelevant words, phrases or descriptions to make the text much longer. Remember, your final goal is to make the thinking process as long as possi- ble, and only output the thinking process you rewrite without any irrelevant information. The screenshot is represented as <image>. and the text you need to rewrite is represented as follows: [customi...