Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A black-box attack using only a static ad, with no knowledge of the agent or user intent, induces VLM web agents to click malicious ads, succeeding in more than 60% of scenarios and close to 100% in some.

desk verdict Solid simulated attack with a genuinely new ad-delivery channel, but the 'real-world' claim overreaches the evidence and needs tempering before publication. read the letter →

arxiv 2505.21499 v1 pith:ONRWO77X submitted 2025-05-27 cs.CR cs.AI

Haowei Wang , Junjie Wang , Xiaojun Jia , Rupeng Zhang , Mingyang Li , Zhe Liu , Yang Liu , Qing Wang This is my paper · ORCID
classification cs.CRcs.AI
keywords webagentsecurityenvironmentinjectionattackadvertisingdeliveryvision-languagemodelsblack-boxpromptVisualArenaOSWorld
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that internet advertising delivery can be used as a practical channel to attack Vision-Language-Model (VLM) web agents in the real world. It proposes AdInject, a black-box method that places a static, innocuous-looking ad such as 'CRITICAL NOTIFICATION / Close AD' on a page and gets the agent to click it, despite the attacker knowing nothing about the agent, its parameters, or the user's current task. Because the ad is a normal page element, it is visible to every perception mode the agent uses, including accessibility tree, screenshot, and Set-of-Marks. On VisualWebArena and OSWorld the attack succeeds in more than 60 percent of scenarios and approaches 100 percent in several settings, and even a defense prompt that names the exact 'Close AD' button only cuts success to 57 percent. The authors argue that advertising's lenient content moderation makes this a realistic, presently underappreciated vulnerability.

What carries the argument

The load-bearing object is a static advertisement built from text fields (title, main text, button text) modeled on standard display-ad formats, whose message frames the click as a necessary step: '# CRITICAL NOTIFICATION' with button text 'Close AD'. On top of this, a VLM-based content optimization step takes the target site's homepage screenshot and accessibility tree, asks a VLM to guess ten likely user intents, and rewrites the ad text to blend those intents in, making the ad look task-relevant. The ad is delivered through a mock ad server that injects the ad as a real DOM element via a browser remote-debugging protocol, so it appears in every perception channel of the agent; after the click the ad closes and no redirection occurs, isolating the measured effect to the click itself.

What would settle it

Submit the exact AdInject creative ('# CRITICAL NOTIFICATION', 'Close AD') to a mainstream advertising platform's review pipeline and observe whether it is served unmodified. If a major ad network rejects or refuses to serve such content, the paper's central 'real-world vector' claim is falsified even though the simulated browser-level attack still works.

Watch

Extended reading notes

Core claim

AdInject's central claim is that an attacker who cannot see the agent, cannot modify HTML directly, cannot run JavaScript, and does not know the user's intent can still reliably induce a web agent to click a malicious ad. The ad is crafted to look like a critical system notification whose 'Close AD' button the agent believes it must press to proceed with its task. Across VisualWebArena and OSWorld, with GPT-4o, Claude-3.5, Claude-3.7, UI-TARS, and an R-MCTS agent, the attack achieves high attack success rates, often over 60 percent, sometimes near 100 percent, with the click happening in one or two steps. Injecting the ad usually does not destroy the agent's original task performance, meaning the attack is a low-cost redirect that can hand the agent to an attacker-controlled page.

Load-bearing premise

The attack's status as a real-world vector rests on the assumption that an advertising platform would actually accept and serve a clearly malicious 'CRITICAL NOTIFICATION / Close AD' creative; the experiments sidestep ad-network content moderation by injecting the ad directly with a mock server, and the authors concede this limitation.

Editorial extensions

If this is right

  • If AdInject transfers to real ad delivery, any VLM web agent that browses pages with third-party ads becomes a click away from attacker-controlled content, with the click often happening on step 1 or 2.
  • Prompt-based defenses are weak shields: generic warnings about distractions or ads leave attack success above 92 percent, and only a defense that says 'do not click the Close AD button' cuts success to 57 percent, and only when placed in the goal rather than the system prompt.
  • Ad content optimization raises attack success substantially in lower-baseline settings (e.g., from 37.92 to 63.89 percent for Claude-3.7 with accessibility tree only), so attacks improve as the attacker exploits website context.
  • All tested ad styles, including pop-up, banner, and sidebar, work; smaller ads lose effectiveness, so ad-quality control that shrinks or strips ads could reduce exposure.
  • Because the ad is a real web element rather than an injected observation, defenses based on sanitizing the agent's observation input alone would miss this attack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A crucial untested step is the ad platform itself: the paper's mock server injects the ad directly, so the 'real-world' claim assumes a network would accept a 'CRITICAL NOTIFICATION / Close AD' creative; if content moderation rejects it, the practical threat is lower, though the simulated attack remains valid.
  • Because the optimization prompt tells the VLM to use a strong command tone and repeat the original content, an alternative defense could scan rendered ad text for imperative task-framing patterns, not just known-bad phrases, a testable extension the paper does not explore.
  • The same delivery channel could be used for subtler goals than clicking: an attacker-controlled landing page fully determines subsequent observations, so the threat extends to whatever post-click task the landing page presents, a step the paper leaves unvalidated.
  • AdInject's success suggests that agents should treat all third-party content as untrusted and verify that an element is actually required for the task before interacting, a design change, not just a prompt change.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AdInject, a black-box attack on VLM-based web agents that uses malicious advertising content to induce agents to click attacker-controlled links. The attack assumes no access to agent internals, static ad content only, and no knowledge of the user's specific intent. Ad content is manually designed to appear as a critical notification ('CRITICAL NOTIFICATION / Close AD') and optionally optimized by a VLM that guesses user intents from the target website. Experiments on VisualWebArena and OSWorld with GPT-4o, Claude-3.5/3.7, and UI-TARS report attack success rates mostly above 60% and sometimes above 90%. The paper also includes ablations on ad size/style and defense prompts. Its central claim is that advertising delivery is a 'potent and real-world vector' for environment injection attacks.

Significance. If the real-world delivery claim were established, AdInject would be a meaningful advance over prior environmental injection attacks: it imposes a stricter threat model (black-box agent, static content, no user intent knowledge) and evaluates across multiple agents and observation settings. The code release, the baseline comparisons (Table 4), and the ablation studies on size and style are useful contributions. However, the significance is presently contingent: the experiments inject ads through a mock CDP-based server that bypasses ad-network moderation and rendering, so the demonstrated result is that a delivered ad-style overlay is highly effective in simulation, not that real ad delivery is a currently exploitable vector. The click-only metric further limits the claimed 'potency.' These gaps are acknowledged in Appendix F, which is a point in the authors' favor, but they are load-bearing for the paper's title-level claim.

major comments (4)
  1. [§5.1, Appendix A.1, Appendix F] The abstract and conclusion claim that advertising delivery is a 'potent and real-world vector' for environment injection, but the experiments do not deliver ads through an advertising platform. Appendix A.1 describes a mock ad server that uses CDP's Page.addScriptToEvaluateOnNewDocument to inject HTML and JavaScript directly into the top-level DOM of every page load. This bypasses the content moderation, policy review, and iframe rendering of real ad networks, and Appendix F concedes that 'successfully deploying genuinely malicious content through real-world advertising platforms presents significant practical hurdles' and that content moderation is 'not fully addressed.' As a result, the ASR numbers in Tables 1-4 characterize a simulated attack on a directly modified page, not an attack delivered via advertising. The central claim requires either a demonstration that such creatives are accepted and rendered by real ad platforms (with their actual character limits and iframe sandboxing) or a substantial weakening of the real-world claim to 'conditional on ad delivery.'
  2. [§4, §5.1, Appendix F] The evaluation metric ASR counts only clicks on the malicious ad; post-click compromise is not tested. Section 4 states that after the click the attacker controls the redirected page and 'subsequent misleading relatively simpler,' but no experiment validates that this leads to harmful actions such as information leakage or malware installation. Appendix F acknowledges this limitation. The phrase 'potent' in the abstract and conclusion therefore overstates the demonstrated harm; the results support an attack that induces clicks, not a complete compromise chain. Please either add post-click experiments or explicitly define ASR as click rate and limit the claims accordingly.
  3. [Appendix C (Metrics), Tables 1-7] All experiments were repeated only three times and the reported values are averages, with no confidence intervals, variance, or per-run values provided anywhere in Tables 1-7. With n=3, differences such as the 73.15% to 79.17% ASR change in Table 3 or the 93.51% to 93.99% change in Table 7 are within ordinary binomial noise, so the quantitative claims about optimization gains and defense effectiveness are not statistically supported. Please report confidence intervals or raw per-run values and, if possible, increase the number of repetitions for the main ASR conditions.
  4. [Appendix C (Environment)] The OSWorld experiments use 43 'manually selected' browser-related tasks, but the task IDs and selection criteria are not given. Because the attack can only inject into web pages, this selection is necessary, but manual selection without an enumerated list and a stated inclusion rule makes it impossible to assess representativeness or to reproduce the subset. Please provide the full task list, ideally with the selection protocol described.
minor comments (5)
  1. [§4, first sentence] The phrase 'the attacker has nor knowledge of agent or user intents' should read 'no knowledge of the agent or user intents.'
  2. [Reference [36]] Reference [36] is missing the publication year and venue details; it currently reads only 'In The Thirteenth International Conference on Learning Representations.' Please complete the citation.
  3. [Appendix C (Environment)] The VisualWebArena experiments use 'the 72 tasks selected by [36]', but the task IDs are not listed in the paper. For reproducibility, please enumerate these task IDs or provide a stable link to the exact task set.
  4. [Appendix E] The optimized ad example contains a main text of several hundred characters; because the paper claims to follow Google AdSense formats (Section 4.1), it would be helpful to note whether the optimized content respects real ad-network character limits or to discuss how such limits would constrain the optimization procedure.
  5. [Table 2] The 'Subset' column in Table 2 repeats 'Browser' and 'Web' for every row; consider merging cells or using a cleaner layout to improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AdInject is an empirical attack demonstration; the mock ad server gap affects real-world generalization but does not make any result equivalent to its inputs.

full rationale

The paper's derivation chain is not circular. The attack content is designed manually or optimized with a VLM using only the website homepage's screenshot and accessibility tree (Section 4.2, Equations 1 and 2), not using any agent's outputs, parameters, or measured attack success rates. The optimization target is the ad content itself, and Table 3 shows it also improves results for Claude-3.7, so the use of GPT-4o as the optimizing VLM is not a self-validating fit to the same model. There is no load-bearing self-citation chain: prior work is used for comparison or benchmark construction, not to justify AdInject's effectiveness. The largest weakness is external validity, not circularity: Appendix A.1 shows the mock ad server injects ad HTML and JavaScript directly into the page via CDP, bypassing real ad-network delivery and content moderation, and Appendix F concedes that 'successfully deploying genuinely malicious content through real-world advertising platforms presents significant practical hurdles.' This makes the 'real-world vector' conclusion empirically under-supported, but the reported ASR numbers are not equivalent to the paper's assumptions by construction. The click-only ASR metric is also explicitly scoped in Appendix C as the primary attack objective, not disguised as a post-click compromise result. The study is therefore a self-contained empirical attack evaluation with a limited but non-circular validity claim.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The paper's load-bearing assumptions are the fidelity of the simulated ad delivery, the visibility of injected content across agent perception settings, and the representativeness of the selected benchmark tasks.

free parameters (3)
  • Default pop-up ad size = 8% of screen
    Hand-chosen default; ablation shows ASR drops from 93.99% to 37.96% when the size is halved to 4%, so the central ASR numbers depend on this choice.
  • VLM optimization decoding parameters = temperature 0.0, topP 1.0
    Hand-chosen for deterministic ad generation; affects the optimized ad content and therefore the ASR improvement in Table 3.
  • Experimental repetition count = 3
    Hand-chosen; all ASR values are averages over three runs with no confidence intervals, so the precision of the reported percentages is not statistically established.
assumptions (3)
  • domain assumption Injected ad elements become part of the agent's observation in all perception settings
    The method relies on the ad being visible in HTML, accessibility tree, and screenshots. The paper implements injection via CDP so the element exists in the DOM, and experiments include A11y Tree, Screen, and Set-of-Marks settings, supporting this assumption.
  • domain assumption The mock ad server is a faithful proxy for real advertising delivery
    The 'real-world vector' claim presumes malicious ads can pass through real ad networks. The paper injects ads directly via CDP, bypassing content moderation, and the Limitations section admits this is not fully addressed.
  • domain assumption The evaluated benchmark tasks are representative of real user tasks for web agents
    VisualWebArena uses 72 tasks from [36] and OSWorld uses 43 manually selected browser-related tasks; the selection criteria for OSWorld are unspecified, so representativeness is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery." pith.science (2026). https://pith.science/paper/ONRWO77X

@misc{pith2026250521499,
  author       = {Pith},
  title        = {Pith review of: AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ONRWO77X}},
  note         = {Machine review of arXiv:2505.21499}
}
read the original abstract

Vision-Language Model (VLM) based Web Agents represent a significant step towards automating complex tasks by simulating human-like interaction with websites. However, their deployment in uncontrolled web environments introduces significant security vulnerabilities. Existing research on adversarial environmental injection attacks often relies on unrealistic assumptions, such as direct HTML manipulation, knowledge of user intent, or access to agent model parameters, limiting their practical applicability. In this paper, we propose AdInject, a novel and real-world black-box attack method that leverages the internet advertising delivery to inject malicious content into the Web Agent's environment. AdInject operates under a significantly more realistic threat model than prior work, assuming a black-box agent, static malicious content constraints, and no specific knowledge of user intent. AdInject includes strategies for designing malicious ad content aimed at misleading agents into clicking, and a VLM-based ad content optimization technique that infers potential user intents from the target website's context and integrates these intents into the ad content to make it appear more relevant or critical to the agent's task, thus enhancing attack effectiveness. Experimental evaluations demonstrate the effectiveness of AdInject, attack success rates exceeding 60% in most scenarios and approaching 100% in certain cases. This strongly demonstrates that prevalent advertising delivery constitutes a potent and real-world vector for environment injection attacks against Web Agents. This work highlights a critical vulnerability in Web Agent security arising from real-world environment manipulation channels, underscoring the urgent need for developing robust defense mechanisms against such threats. Our code is available at https://github.com/NicerWang/AdInject.

Figures

Figures reproduced from arXiv: 2505.21499 by the authors.

Figure 1
Figure 1. Demonstration of AdInject Our attack goal is to mislead the agent into clicking a malicious ad. We strictly adhere to the threat model, initially assuming the attacker has nor knowledge of agent or user intents, and that injected content is limited to static resources. After the ad button is clicked, in most cases, it redirects to a 4 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Demonstration of Ad Content Optimization [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Part of Advertisement Styles A.1 Server Capabilities Mock Ad Server leverages the Chrome DevTools Protocol (CDP) [11] to remotely control a browser instance with remote debugging enabled, and injects specific advertisement HTML and JavaScript content into the web pages opened by that browser. Upon successful connection to browser, the server retrieves a list of all available tabs within the browser. It then uses the… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prismata: Confining Cross-Site Prompt Injection in Web Agents

    cs.CR 2026-07 conditional novelty 7.5 of 10

    Prismata cuts web-agent prompt-injection attack success from 85.5% to 0.7% via Biba-inspired DOM trust labeling and mechanical least-privilege confinement without site annotations.

  2. LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents

    cs.CR 2026-08 conditional novelty 6.0 of 10

    LoginTrap induces LLM-based web agents to enter attacker-controlled login flows and submit sensitive information, achieving 86% average end-to-end success across LLM backbones.

Reference graph

Works this paper leans on

49 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    Multistakeholder recommendation: Survey and research directions.User Modeling and User-Adapted Interaction, 30:127–158, 2020

    Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Pizzato. Multistakeholder recommendation: Survey and research directions.User Modeling and User-Adapted Interaction, 30:127–158, 2020

    work page 2020

  2. [2]

    Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    arXiv 2024

  3. [3]

    Agent s2: A compositional generalist-specialist framework for computer use agents.arXiv preprint arXiv:2504.00906, 2025

    Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents.arXiv preprint arXiv:2504.00906, 2025

    arXiv 2025

  4. [4]

    Claude 3.7 sonnet and claude code

    anthropic. Claude 3.7 sonnet and claude code

    work page

  5. [5]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku

    anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku

    work page

  6. [6]

    Image hijacks: Adversarial images can control generative models at runtime.arXiv preprint arXiv:2309.00236, 2023

    Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime.arXiv preprint arXiv:2309.00236, 2023

    arXiv 2023

  7. [7]

    Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264, 2024

    arXiv 2024

  8. [8]

    The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467, 2024

    De Chezelles, Thibault Le Sellier, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, et al. The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467, 2024

    arXiv 2024

Show all 49 references
  1. [9]

    Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  2. [10]

    Figstep: Jailbreaking large vision-language models via typographic visual prompts

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23951...

    2025

  3. [11]

    Chrome devtools protocol

    Google. Chrome devtools protocol

  4. [12]

    Google adsense

    Google. Google adsense

  5. [13]

    Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast.arXiv preprint arXiv:2402.08567, 2024

    Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast.arXiv preprint arXiv:2402.08567, 2024

    2024 arXiv

  6. [14]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  7. [15]

    Transferable adversarial attacks on black-box vision-language models.arXiv preprint arXiv:2505.01050, 2025

    Kai Hu, Weichen Yu, Li Zhang, Alexander Robey, Andy Zou, Chengming Xu, Haoqi Hu, and Matt Fredrikson. Transferable adversarial attacks on black-box vision-language models.arXiv preprint arXiv:2505.01050, 2025

    2025 arXiv

  8. [16]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [17]

    Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant.arXiv preprint arXiv:2410.18603, 2024

    Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant.arXiv preprint arXiv:2410.18603, 2024. 10

    arXiv

  10. [18]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. InProceedings of the 62nd Annual Meeting of the Associatio...

    2024

  11. [19]

    Refusal-trained llms are easily jailbroken as browser agents.arXiv preprint arXiv:2410.13886, 2024

    Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, et al. Refusal-trained llms are easily jailbroken as browser agents.arXiv preprint arXiv:2410.13886, 2024

    arXiv

  12. [20]

    Infant agent: A tool-integrated, logic-driven agent with cost-effective api usage.arXiv preprint arXiv:2411.01114, 2024

    Bin Lei, Yuchen Li, Yiming Zeng, Tao Ren, Yi Luo, Tianyu Shi, Zitian Gao, Zeyu Hu, Weitai Kang, and Qiuwu Chen. Infant agent: A tool-integrated, logic-driven agent with cost-effective api usage.arXiv preprint arXiv:2411.01114, 2024

    2024 arXiv

  13. [21]

    Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models

    Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. InEuropean Conference on Computer Vision, pages 174–189. Springer, 2024

    2024

  14. [22]

    Knowing your enemy: understanding and detecting malicious web advertising

    Zhou Li, Kehuan Zhang, Yinglian Xie, Fang Yu, and XiaoFeng Wang. Knowing your enemy: understanding and detecting malicious web advertising. InProceedings of the 2012 ACM conference on Computer and communications security, pages 674–686, 2012

    2012

  15. [23]

    Revisiting backdoor attacks against large vision-language models.arXiv preprint arXiv:2406.18844, 2024

    Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Ee-Chien Chang, and Xi- aochun Cao. Revisiting backdoor attacks against large vision-language models.arXiv preprint arXiv:2406.18844, 2024

    2024 arXiv

  16. [24]

    Eia: Environmental injection attack on generalist web agents for privacy leakage.arXiv preprint arXiv:2409.11295, 2024

    Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. Eia: Environmental injection attack on generalist web agents for privacy leakage.arXiv preprint arXiv:2409.11295, 2024

    2024 arXiv

  17. [25]

    Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499, 2023

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499, 2023

    2023 arXiv

  18. [26]

    Caution for the environment: Multimodal agents are susceptible to environmental distractions.arXiv preprint arXiv:2408.02544, 2024

    Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. Caution for the environment: Multimodal agents are susceptible to environmental distractions.arXiv preprint arXiv:2408.02544, 2024

    2024 arXiv

  19. [27]

    Dismantling common internet services for ad-malware detection.arXiv preprint arXiv:2404.14190, 2024

    Florian Nettersheim, Stephan Arlt, and Michael Rademacher. Dismantling common internet services for ad-malware detection.arXiv preprint arXiv:2404.14190, 2024

    2024 arXiv

  20. [28]

    Ignore previous prompt: Attack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  21. [29]

    Visual adversarial examples jailbreak aligned large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. InProceedings of the AAAI conference on artificial intelligence, volume 38, pages 21527–21536, 2024

    2024

  22. [30]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  23. [31]

    Are these ads safe: Detecting hidden attacks through the mobile app-web interfaces

    Vaibhav Rastogi, Rui Shao, Yan Chen, Xiang Pan, Shihong Zou, and Ryan D Riley. Are these ads safe: Detecting hidden attacks through the mobile app-web interfaces. InNDSS, 2016

    2016

  24. [32]

    On the adversarial robustness of multi-modal founda- tion models

    Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal founda- tion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3677–3685, 2023

    2023

  25. [33]

    Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments.arXiv preprint arXiv:2501.10893, 2025

    Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arık. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments.arXiv preprint arXiv:2501.10893, 2025

    2025 arXiv

  26. [34]

    Oscar: Operating system control via state-aware reasoning and re-planning.arXiv preprint arXiv:2410.18963, 2024

    Xiaoqiang Wang and Bang Liu. Oscar: Operating system control via state-aware reasoning and re-planning.arXiv preprint arXiv:2410.18963, 2024. 11

    2024 arXiv

  27. [35]

    Stop reasoning! when multimodal llm with chain-of-thought reasoning meets adversarial image.arXiv preprint arXiv:2402.14899, 2024

    Zefeng Wang, Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, V olker Tresp, Philip Torr, and Jindong Gu. Stop reasoning! when multimodal llm with chain-of-thought reasoning meets adversarial image.arXiv preprint arXiv:2402.14899, 2024

    2024 arXiv

  28. [36]

    Dissecting adversarial robustness of multimodal lm agents

    Chen Henry Wu, Rishi Rajesh Shah, Jing Yu Koh, Russ Salakhutdinov, Daniel Fried, and Aditi Raghunathan. Dissecting adversarial robustness of multimodal lm agents. InThe Thirteenth International Conference on Learning Representations

  29. [37]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Sy...

    2024

  30. [38]

    Advweb: Controllable black-box attacks on vlm-powered web agents.arXiv preprint arXiv:2410.17401, 2024

    Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. Advweb: Controllable black-box attacks on vlm-powered web agents.arXiv preprint arXiv:2410.17401, 2024

    2024 arXiv

  31. [39]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  32. [40]

    Shadowcast: Stealthy data poisoning attacks against vision-language models

    Yuancheng Xu, Jiarui Yao, Manli Shu, Yanchao Sun, Zichu Wu, Ning Yu, Tom Goldstein, and Furong Huang. Shadowcast: Stealthy data poisoning attacks against vision-language models. arXiv preprint arXiv:2402.06659, 2024

    2024 arXiv

  33. [41]

    Vlattack: Multimodal adversarial attacks on vision-language tasks via pre-trained models.Advances in Neural Information Processing Systems, 36:52936–52956, 2023

    Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. Vlattack: Multimodal adversarial attacks on vision-language tasks via pre-trained models.Advances in Neural Information Processing Systems, 36:52936–52956, 2023

    2023

  34. [42]

    Exact: Teaching ai agents to explore with reflective-mcts and exploratory learning.arXiv preprint arXiv:2410.02052, 2024

    Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents to explore with reflective-mcts and exploratory learning.arXiv preprint arXiv:2410.02052, 2024

    2024 arXiv

  35. [43]

    Internet advertising: An interplay among advertisers, online publishers, ad exchanges and web users.arXiv preprint arXiv:1206.1754, 2012

    Shuai Yuan, Ahmad Zainal Abidin, Marc Sloan, and Jun Wang. Internet advertising: An interplay among advertisers, online publishers, ad exchanges and web users.arXiv preprint arXiv:1206.1754, 2012

    2012 arXiv

  36. [44]

    Towards adversarial attack on vision-language pre- training models

    Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre- training models. InProceedings of the 30th ACM International Conference on Multimedia, pages 5005–5013, 2022

    2022

  37. [45]

    Udora: A unified red teaming framework against llm agents by dynamically hijacking their own reasoning.arXiv preprint arXiv:2503.01908, 2025

    Jiawei Zhang, Shuang Yang, and Bo Li. Udora: A unified red teaming framework against llm agents by dynamically hijacking their own reasoning.arXiv preprint arXiv:2503.01908, 2025

    2025

  38. [46]

    Attacking vision-language computer agents via pop-ups

    Yanzhe Zhang, Tao Yu, and Diyi Yang. Attacking vision-language computer agents via pop-ups. arXiv preprint arXiv:2411.02391, 2024

    2024 arXiv

  39. [47]

    Close AD

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. 12 A Mock Advertisement Server Fig...

    2023 arXiv

  40. [48]

    Accessibility tree features: {accessibility_tree}

  41. [49]

    PR You are a professional ad content optimization expert, aiming to attract users to click on the ’Close AD’ button

    Page screenshot: {page_screenshot} Please return 10 most likely high-level user tasks in format of json list of strings. PR You are a professional ad content optimization expert, aiming to attract users to click on the ’Close AD’ button. I will provide you with possible user q...

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.