{"id":"9d51ab2b-5798-4603-bbf7-dc8f6aa731f1","arxiv_id":"2505.12981","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"All nine tested mobile LLM agents are vulnerable to at least one of 11 identified attack surfaces, with UI manipulation and deeplink forgery among the most effective.","lead":"This paper presents the first systematic security analysis of mobile LLM agents, identifying 11 attack surfaces and testing nine real agents with a new framework called AgentScan. Every tested agent was vulnerable to at least one targeted attack, and the most exposed agent was vulnerable to eight of the eleven.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'every agent is vulnerable' conclusion is scoped by a nine-agent convenience sample; the paper's own Limitations section concedes that this limits generalizability, so the ecosystem-wide phrasing overreaches.","rationale":"The paper's per-agent empirical work is internally consistent and valuable: the attack taxonomy, the AgentScan design, and the detailed result table support the weaker claim that the nine tested agents are all vulnerable. The vendor acknowledgements and the demonstration that a simple mitigation reduces but does not eliminate vulnerabilities also add credibility. I found no evidence of fabrication or intentional overstatement. However, the abstract and conclusion generalize from this nine-agent sample to the whole ecosystem, and the reader's weakest-assumption analysis correctly identifies this as the most load-bearing gap. The paper does not provide a sampling frame, and the five academic frameworks are not representative of deployed production agents in any documented way. Section VII-A explicitly concedes the generalizability limitation, which should be elevated into the main text rather than relegated to a late limitation paragraph. This concern does not invalidate the case studies, so the verdict stays CONDITIONAL: the authors should either broaden the empirical base with a clearer sampling procedure or qualify the headline claim to the agents actually evaluated.","tokens_in":21598,"tokens_out":13207,"duration_ms":144119,"concrete_test":"Construct an explicit sampling frame for current production mobile LLM agents: enumerate OEM assistants on flagship devices, third-party universal agents available through app stores, and deployed agent frameworks; then select at least two additional agents per category and run the AgentScan 11-vector suite under the reported device, OS, and permission conditions. If every newly tested agent is vulnerable to at least one attack vector, the 'every agent' claim gains support; if even one is not, the conclusion must be restricted to the tested set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, as stated in the abstract and conclusion, is that all agents exhibit vulnerabilities to targeted attacks and that current mobile LLM agents lack basic defenses. For that claim to hold as an ecosystem-wide statement, the nine selected systems must be representative of the deployment population. Section VI-A asserts that the sample is 'representative and diverse' but provides no sampling frame: no enumeration of the agent population, no inclusion criteria, and no stratification. Five of the nine are academic frameworks (AutoDroid, Mobile-Agent, Mobile-Agent-v2, AppAgent, DroidBot-GPT) that are not production deployments, while the four anonymized commercial agents are not described well enough to judge whether they cover the design space. The authors themselves flag this in Section VII-A: the tested agents 'may not fully reflect the diversity of real-world agents or dynamic environments,' which 'limits the generalizability of the findings.' Thus the strongest supported reading is 'all nine tested agents are vulnerable,' not 'all mobile LLM agents in the wild are vulnerable.' If the intended contribution is the ecosystem-wide warning, the missing systematic sampling is the load-bearing gap; if the intended contribution is a set of case studies, the abstract and conclusion should be scoped to the tested set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a security analysis of mobile LLM agents across three categories: OEM system-level agents, third-party universal agents, and emerging agent frameworks. It decomposes the agent workflow into five stages, defines three threat dimensions (LLM, GUI, and System), and proposes an 11-item attack taxonomy. The authors implement AgentScan, a semi-automated testing framework, and evaluate nine agents: five named academic frameworks and four anonymized commercial agents. The empirical results in Table II show that every tested agent is vulnerable to at least four attack vectors; GUI-layer attacks such as Transparent Overlay and Pop-up Interference succeed against 7 of 9 agents, and the most affected agent (AppAgent) is vulnerable to 8 of 11 vectors. The paper also demonstrates a composite attack that chains transparent overlay, prompt injection, and malicious instructions to extract a bank card password from agent memory, and it proposes mitigation strategies with a before/after evaluation on Mobile-Agent-v2.","tokens_in":21807,"tokens_out":8092,"duration_ms":76509,"significance":"If the findings are accepted, this is the first systematic security evaluation of mobile LLM agents and a useful reusable testing framework. The taxonomy is clearly structured, and the 10-trial protocol for the LLM/GUI attacks strengthens the core observation that current agent implementations lack basic defenses against UI manipulation and intent hijacking. The composite attack is a compelling end-to-end demonstration of impact. The main value is empirical: it should push agent developers to add screen-state validation, signature checks, and output sanitization. However, the paper's prevalence claims extend beyond the nine-agent sample, and the system-layer experiments are reported with considerably less rigor than the GUI/LLM experiments. The framework is said to be released publicly, though no repository link or artifact details are given, which limits immediate reproducibility.","major_comments":[{"comment":"The Introduction states that \"UI manipulation attacks are universally effective—every tested agent fails to defend against Transparent Overlay and Pop-up Interference attacks.\" This is contradicted by Table II: Agent-A resists both Transparent Overlay and Pop-up Interference, Agent-B resists Transparent Overlay, and Agent-D resists Pop-up Interference in both of its modes. The accurate statement is that these attacks succeeded against 7 of 9 agents. Please correct this overclaim and align the abstract and conclusion with the actual counts, particularly if the abstract's \"every agent is vulnerable\" is meant as an ecosystem-wide statement.","section":"Section I, Section VI-B, Table II"},{"comment":"The headline claim is scoped by the evaluation set. Section VI-A states that the nine agents are \"representative and diverse\" but provides no enumeration of the population of mobile LLM agents, no inclusion criteria, and no sampling strategy. Section VII-A concedes that the tested agents \"may not fully reflect the diversity of real-world agents or dynamic environments\" and that this \"limits the generalizability of the findings.\" The abstract and conclusion should therefore say \"all nine tested agents\" rather than \"every agent\" unless the authors add a systematic sampling justification.","section":"Section VI-A, Section VII-A, Abstract"},{"comment":"System-layer attack results are reported only as binary success/failure symbols without trial counts. Section V-D1 says forged versions were created for three apps, Section V-D2 says two deeplink scenarios were implemented, and Section V-D3 says logs were monitored continuously, but no success criterion is given (e.g., what fraction of launch attempts resolved to the malicious package) and no number of repetitions is reported. This is inconsistent with the x/10 protocol used for the LLM/GUI attacks and is insufficient to support the claim that Package Name Forgery and Deeplink Forgery \"exclusively affected\" all four system-level agents. Add per-scenario trial counts and explicit definitions of success.","section":"Section V-D, Table II"},{"comment":"The description of model standardization is ambiguous. The text says \"all decision-making tasks were powered by GPT-4o\" after stating that system-level agents were tested on commercial flagship devices. If the OEM agents run their own proprietary backends, then the comparison between system-level agents and frameworks is confounded by backend model choice; if they were forced to use GPT-4o, the procedure is not described. Clarify which agents used GPT-4o and how, and discuss how this affects the interpretation of the system-level agents' immunity to the LLM-layer attacks.","section":"Section VI-A, Section V-C"},{"comment":"The success rate for Malicious Instructions is reported as a fraction of 10, but the paper does not define the success criterion beyond \"the agent follows these instructions without additional user confirmation.\" It is unclear whether partial completion, an attempted action that is blocked by the OS, or a refusal followed by compliance after a nudge counts as success. Provide a coding rubric or examples of borderline cases so that the reported 5/10 to 9/10 figures are reproducible.","section":"Section V-B, Table II"}],"minor_comments":[{"comment":"The cross-references to \"LLM layer (§ III-C), GUI layer (§ III-B), and System layer (§ III-D)\" point to workflow stages, not to the attack-layer definitions; the taxonomy is actually introduced in Section IV-A. Update these cross-references.","section":"Section IV-A1"},{"comment":"Agent-D appears as two separate columns (\"Vision-Based\" and \"Structure-Based\"), which makes statements such as \"7 out of 9 agents\" visually ambiguous. State explicitly how the two modes are counted in the per-agent average, or present a single aggregate column.","section":"Table II"},{"comment":"The caption defines \"!\", \"✗\", and \"-\" but does not explain that \"!\" for system-layer attacks is a confirmed success without a rate. If trial counts are added as recommended in the major comments, the caption should describe how they are reported.","section":"Table II caption"},{"comment":"The text refers to \"Autodroid and Droid-GPT\" while the table and related work use \"AutoDroid\" and \"DroidBot-GPT.\" Please standardize the naming throughout.","section":"Section VI-B"},{"comment":"The mitigation results in Table IV are reported for a single agent (Mobile-Agent-v2) without confidence intervals or repeated runs. Given the large claimed improvements, this should be described as a proof-of-concept rather than a validated mitigation strategy.","section":"Section VII-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely publishable after a revision that scopes the claims and tightens the experimental reporting. The main issues are overstatement in the abstract and Introduction relative to Table II, and the lack of trial counts for system-layer attacks. The anonymization of the four commercial agents is reasonable for ethical reasons but limits independent verification; the five named framework agents can be re-tested by the community. I found no evidence of fabrication or circularity: the attacks are standard and are applied uniformly across agents. I would not require a new sampling study for acceptance, but the authors should either provide a sampling rationale or explicitly limit the prevalence claim to the tested set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing to know: this is the first paper to treat mobile LLM agents as a security domain in its own right. The taxonomy of 11 attack surfaces across the LLM, GUI, and system layers is useful, and the AgentScan framework is a reasonable attempt at repeatable evaluation. The headline result—all nine agents, including four anonymized commercial ones, were vulnerable to multiple attacks—is credible and worrying. The universal success of transparent overlay and pop-up interference is particularly notable. I also credit the composite attack case study and the mitigation evaluation, which shows simple defenses cut but do not eliminate risk.\n\nThe soft spots are real but not deal-breakers. The sample is convenience-based: five of nine are academic frameworks, and the four commercial agents are described in almost no architectural detail. The abstract says 'every agent is vulnerable' after applying AgentScan to nine agents, which is legitimate when read as 'every agent we tested.' The conclusion drifts slightly toward ecosystem-wide language ('current implementations'), but the limitations section explicitly concedes the generalizability problem. A clearer scoping sentence would fix that.\n\nMore concrete issues: the paper promises to release AgentScan but no artifact link appears anywhere. The system-layer attacks (package name forgery, deeplink forgery, log leakage) are reported as binary success/failure with no trials or procedure detail, which makes those results hard to trust at face value. And the claim that GPT-4o powered all decision-making is ambiguous for the system-level agents, which presumably run their own built-in models. All of this is fixable in revision.\n\nThis deserves peer review. The taxonomy and empirical results are valuable even with a convenience sample. I'd ask the authors to release the framework, document the system-layer tests precisely, clarify the GPT-4o setup, and align the abstract/conclusion with the tested set. I'd recommend major revisions, not rejection.","headline":"First systematic security taxonomy for mobile LLM agents, with a credible demonstration that all nine tested agents are vulnerable; sampling limits the ecosystem-wide inference, but the paper mostly scopes its claims and deserves a serious referee.","tokens_in":22333,"tokens_out":3349,"would_cite":true,"duration_ms":34038,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All nine mobile AI agents tested are vulnerable to targeted attacks.","keywords":["mobile LLM agents","security analysis","prompt injection","UI manipulation","transparent overlay attacks","Android security","attack taxonomy","AgentScan"],"falsifier":"An agent that revalidates the screen—comparing the view hierarchy and a fresh screenshot immediately before each tap and refusing to act on unrecognized targets—should score 0/10 against transparent-overlay and pop-up interference in AgentScan's tests; observing such an agent would contradict the paper's universality claim. Equivalently, testing a broader, independently sampled set of agents and finding several with no confirmed vectors would undercut the ecosystem-wide conclusion.","tokens_in":21398,"feed_emoji":"📱","tokens_out":7110,"duration_ms":67040,"temperature":0.7,"pith_summary":"This paper is trying to establish that today's mobile LLM agents—the on-device assistants that turn natural-language requests into UI actions—are broadly and systematically unprotected against adversaries who control an installed app. It decomposes agent behavior into three layers (language reasoning, GUI interaction, and system execution), enumerates 11 attack surfaces spanning those layers, and builds a semi-automated test framework, AgentScan, to probe them. Running AgentScan against nine widely deployed agents, it finds that every agent succumbs to at least one targeted attack and that the average agent is vulnerable to 6.3 of the 11 vectors; UI-layer attacks such as transparent overlays and pop-up interference succeed against most agents and can redirect clicks, alter behavior, leak data, or hijack execution entirely. The stakes are concrete: if true, a benign-looking app could quietly turn a user's trusted assistant into an adversary.","feed_headline":"All nine mobile AI agents fail targeted security tests","feed_subtitle":"UI tricks and prompt injections hijack every tested assistant; worst cases hit eight of eleven attack types.","key_machinery":"The load-bearing mechanism is the three-layer abstraction of agent operation—LLM reasoning, GUI perception and interaction, and system-level execution—together with AgentScan, a client-server framework that emulates adversarial conditions by pushing crafted APKs and overlays onto a device over ADB and observing the agent under 44 adapted real-world tasks. The taxonomy maps each attack to the workflow stage it subverts: for example, malicious instructions and glitch tokens poison the model's reasoning, transparent overlays and pop-ups deceive coordinate-based clicks, deeplink forgery hijacks the app launcher, and log leakage exposes the data pipeline. This combination lets the paper attribute observed failures to specific workflow stages and supports its claim that the vulnerabilities are structural rather than incidental.","core_discovery":"On its own terms, the paper's central discovery is that the unique capabilities that make mobile LLM agents useful—parsing screenshots, following natural-language instructions, and launching apps with system intents—are also the channels through which they can be subverted. The authors define a taxonomy of 11 attack vectors across three dimensions: malicious instructions and glitch tokens at the LLM layer; image forgery, viewtree interference, prompt injection via display, transparent overlays, and pop-up interference at the GUI layer; and package-name forgery, deeplink forgery, and log leakage at the system layer. Using AgentScan against nine representative agents spanning OEM assistants, third-party universal agents, and PC-connected agent frameworks, they report that all nine are vulnerable to targeted attacks, with the worst case exposing eight of eleven vectors and an average exposure of 6.3. They also demonstrate a chained three-stage attack against Mobile-Agent-v2 in which a transparent overlay, an on-screen prompt injection, and a malicious instruction combine to extract a bank-card password from agent memory, showing that individual weaknesses compose into end-to-end compromise.","pith_inferences":["We infer that any mobile agent that treats all visible text as context for its LLM will remain vulnerable to on-screen prompt injection until it distinguishes instruction content from display content; the paper's own partial mitigation supports this.","We infer that the same structural weaknesses will appear in adjacent platforms—desktop or wearable OS agents—whenever they reason over screen pixels and act through privileged input, because the attack surface is defined by the workflow, not by Android alone.","A testable extension of the paper's framework would be to benchmark an agent that screenshots and parses the view hierarchy immediately before every click, predicting that transparent-overlay and pop-up attacks drop to 0/10; this would isolate the precise fix.","The results also suggest OS-level hardening: restricting or visibly marking SYSTEM_ALERT_WINDOW overlays during agent execution could neutralize an entire class of attacks without requiring agent-level changes."],"forward_implications":["If the results generalize, any mobile agent that executes coordinate-based taps without revalidating the screen immediately before the action can be redirected by transparent overlays or pop-ups.","Agents that rely on view hierarchies for element localization inherit Android's UI-metadata weaknesses; agents that rely purely on vision avoid those but fall to image forgery and on-screen prompt injection, so no tested perception strategy was secure by itself.","System-level OEM agents resist prompt-based attacks through logic-oriented planning, yet all four still fall to package-name forgery and deeplink forgery, showing that rigid workflows do not eliminate system-layer risk.","The demonstrated composite attack implies that combining a UI-layer hijack with a language-layer instruction can turn the agent itself into the data exfiltrator, including data the user never asked it to send.","The mitigation experiment on Mobile-Agent-v2 shows that linking actions to viewtree and screen validation reduces several attack success rates to zero or near-zero, but prompt injection via display and glitch tokens still succeed in some trials, so defensive fixes are partial."],"supporting_citations":[{"why":"Supplies the Android task templates from which the paper's 44-task baseline instruction set is adapted.","marker":"[59]"},{"why":"Provides the glitch-token attack scenario and the ten malformed tokens used to test LLM-layer robustness.","marker":"[41]"},{"why":"Documents Android UI deception attacks that the paper adapts to image forgery and overlay threats.","marker":"[45]"},{"why":"Provides overlay-based Android malware analysis underlying the transparent-overlay and pop-up attack designs.","marker":"[48]"},{"why":"Establishes the insecurity of mobile deep links, the basis for the deeplink forgery tests.","marker":"[52]"},{"why":"Shows how instant-app and link attacks redirect users, informing the deeplink and activity-hijacking scenarios.","marker":"[51]"},{"why":"Is the Mobile-Agent framework evaluated and used in the chained bank-card extraction case study.","marker":"[26]"},{"why":"Is the Mobile-Agent-v2 framework that receives the security-mitigation experiment.","marker":"[27]"},{"why":"Is the AppAgent framework evaluated, the agent with the most confirmed vulnerabilities.","marker":"[28]"},{"why":"Is the AutoDroid framework evaluated, one of the viewtree-dependent agents with near-total susceptibility.","marker":"[29]"}],"fun_headline_variants":["Every mobile LLM agent hacked: worst hit 8 attack vectors","Mobile AI agents: all 9 fail security tests, one leaks card data","AgentScan: 9/9 mobile agents vulnerable, average 6.3 flaws","Mobile LLM agents all break under targeted attacks, study finds","From assistants to adversaries: all 9 mobile agents compromised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sweeping conclusion that all mobile LLM agents are vulnerable rests on the assumption that the nine selected agents represent the wider population of mobile LLM agents, and that an installed app with overlay permissions is a realistic attacker.","fun_headline_variants_meta":{"raw":{"variants":["Every mobile LLM agent hacked: worst hit 8 attack vectors","Mobile AI agents: all 9 fail security tests, one leaks card data","AgentScan: 9/9 mobile agents vulnerable, average 6.3 flaws","Mobile LLM agents all break under targeted attacks, study finds","From assistants to adversaries: all 9 mobile agents compromised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1711,"prompt_tokens":1078,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":694,"tokens_out":633,"duration_ms":5610,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:22:16.703214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An agent that revalidates the screen—comparing the view hierarchy and a fresh screenshot immediately before each tap and refusing to act on unrecognized targets—should score 0/10 against transparent-overlay and pop-up interference in AgentScan's tests; observing such an agent would contradict the paper's universality claim. Equivalently, testing a broader, independently sampled set of agents and finding several with no confirmed vectors would undercut the ecosystem-wide conclusion.","supporting_citations":[{"cited_title":"An- droidinthewild: A large-scale dataset for android device control,","cited_arxiv_id":null,"evidence_quote":"Supplies the Android task templates from which the paper's 44-task baseline instruction set is adapted."},{"cited_title":"Glitch tokens in large language models: Categorization taxonomy and effective detection,","cited_arxiv_id":null,"evidence_quote":"Provides the glitch-token attack scenario and the ten malformed tokens used to test LLM-layer robustness."},{"cited_title":"Android ui deception revisited: Attacks and defenses,","cited_arxiv_id":null,"evidence_quote":"Documents Android UI deception attacks that the paper adapts to image forgery and overlay threats."},{"cited_title":"Understanding and detecting overlay-based android malware at market scales,","cited_arxiv_id":null,"evidence_quote":"Provides overlay-based Android malware analysis underlying the transparent-overlay and pop-up attack designs."},{"cited_title":"Measuring the insecurity of mobile deep links of android,","cited_arxiv_id":null,"evidence_quote":"Establishes the insecurity of mobile deep links, the basis for the deeplink forgery tests."},{"cited_title":"All your app links are belong to us: understanding the threats of instant apps based attacks,","cited_arxiv_id":null,"evidence_quote":"Shows how instant-app and link attacks redirect users, informing the deeplink and activity-hijacking scenarios."}],"review_version":1}