{"id":"446341a9-79c0-4201-8fd8-5695e782abe1","arxiv_id":"2505.21499","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fake 'Close AD' ads make VLM web agents click them over 60% of the time, and near 100% in some settings.","lead":"This paper shows that a fake advertisement placed on a web page can trick AI web agents into clicking it, succeeding in most tests and nearly always in some. It is a warning that the ad ecosystem could become a real attack channel against autonomous browsing agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'real-world vector' claim is unsupported: the attack ad is injected into the top-level DOM by a mock CDP server, never passing a real ad network's policy review or iframe rendering; the paper's own Limitations concede this, so current results demonstrate a simulated attack, not an…","rationale":"Reading the paper as an attack on the simulated environment, the work is solid: the threat model is stricter than prior work, the baseline comparison in Table 4 shows near-zero ASR for vanilla and generic injection ads, and the ablations are informative. The problem is the generalization from 'we can inject an element into the DOM via CDP' to 'prevalent advertising delivery constitutes a potent and real-world vector'. The mock server is not an advertising platform: it has no content policy, no auction, no iframe sandbox, and no publisher constraints, and every experiment inherits this. The authors acknowledge the delivery gap in Appendix F, but the abstract and conclusion still assert the real-world vector. This is not an internal inconsistency in the simulated attack; it is a mismatch between the evidence and the claim. Because the condition is testable and the paper itself conditions its conclusion, the reader's CONDITIONAL verdict is appropriate; validating the delivery pipeline or softening the real-world claim would resolve the issue.","tokens_in":14007,"tokens_out":7037,"duration_ms":78802,"concrete_test":"Submit the exact attack creative (title, main text, button, and a neutral landing URL) to at least one major ad platform's review flow using a test or sandbox account, and record approval or disapproval plus the actual rendering container (top-level DOM vs cross-origin iframe). If approved, serve the creative through the real ad-serving path on the VisualWebArena pages and re-measure ASR with the same agents; if the creative is rejected, or if the iframe-rendered ASR is materially below the top-level-DOM mock result, the real-world vector claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that advertising delivery is a potent real-world injection vector requires two conditions the experiments do not test: (1) a real ad platform will accept the 'CRITICAL NOTIFICATION / Close AD' creative, and (2) the delivered ad renders in the same observation space as the mock injection. Appendix A.1 shows the mock server uses CDP's Page.addScriptToEvaluateOnNewDocument to inject HTML and JavaScript directly into the top-level DOM before every page load. That is direct environment manipulation, not ad-network delivery. Real display ads are served in cross-origin iframes and are subject to content policies; creatives that imitate system notifications or claim to obstruct content are exactly the deceptive formats ad networks moderate. The cited evidence for lenient moderation ([1,31,22,27]) is dated and does not address current platform enforcement. The paper's own Limitations (Appendix F) concede that 'deploying genuinely malicious content through real-world advertising platforms presents significant practical hurdles' and that content moderation is 'not fully addressed'. If such creatives are rejected or rendered in a sandboxed iframe that the agent cannot parse, the ASR numbers in Tables 1-4 measure a simulated attack on a directly modified page, not an attack via advertising delivery. A secondary concern is that ASR is click-only; post-click compromise is not validated (also in Appendix F), so 'potent' overstates demonstrated harm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdInject, a black-box attack on VLM-based web agents that uses malicious advertising content to induce agents to click attacker-controlled links. The attack assumes no access to agent internals, static ad content only, and no knowledge of the user's specific intent. Ad content is manually designed to appear as a critical notification ('CRITICAL NOTIFICATION / Close AD') and optionally optimized by a VLM that guesses user intents from the target website. Experiments on VisualWebArena and OSWorld with GPT-4o, Claude-3.5/3.7, and UI-TARS report attack success rates mostly above 60% and sometimes above 90%. The paper also includes ablations on ad size/style and defense prompts. Its central claim is that advertising delivery is a 'potent and real-world vector' for environment injection attacks.","tokens_in":14301,"tokens_out":5944,"duration_ms":62247,"significance":"If the real-world delivery claim were established, AdInject would be a meaningful advance over prior environmental injection attacks: it imposes a stricter threat model (black-box agent, static content, no user intent knowledge) and evaluates across multiple agents and observation settings. The code release, the baseline comparisons (Table 4), and the ablation studies on size and style are useful contributions. However, the significance is presently contingent: the experiments inject ads through a mock CDP-based server that bypasses ad-network moderation and rendering, so the demonstrated result is that a delivered ad-style overlay is highly effective in simulation, not that real ad delivery is a currently exploitable vector. The click-only metric further limits the claimed 'potency.' These gaps are acknowledged in Appendix F, which is a point in the authors' favor, but they are load-bearing for the paper's title-level claim.","major_comments":[{"comment":"The abstract and conclusion claim that advertising delivery is a 'potent and real-world vector' for environment injection, but the experiments do not deliver ads through an advertising platform. Appendix A.1 describes a mock ad server that uses CDP's Page.addScriptToEvaluateOnNewDocument to inject HTML and JavaScript directly into the top-level DOM of every page load. This bypasses the content moderation, policy review, and iframe rendering of real ad networks, and Appendix F concedes that 'successfully deploying genuinely malicious content through real-world advertising platforms presents significant practical hurdles' and that content moderation is 'not fully addressed.' As a result, the ASR numbers in Tables 1-4 characterize a simulated attack on a directly modified page, not an attack delivered via advertising. The central claim requires either a demonstration that such creatives are accepted and rendered by real ad platforms (with their actual character limits and iframe sandboxing) or a substantial weakening of the real-world claim to 'conditional on ad delivery.'","section":"§5.1, Appendix A.1, Appendix F"},{"comment":"The evaluation metric ASR counts only clicks on the malicious ad; post-click compromise is not tested. Section 4 states that after the click the attacker controls the redirected page and 'subsequent misleading relatively simpler,' but no experiment validates that this leads to harmful actions such as information leakage or malware installation. Appendix F acknowledges this limitation. The phrase 'potent' in the abstract and conclusion therefore overstates the demonstrated harm; the results support an attack that induces clicks, not a complete compromise chain. Please either add post-click experiments or explicitly define ASR as click rate and limit the claims accordingly.","section":"§4, §5.1, Appendix F"},{"comment":"All experiments were repeated only three times and the reported values are averages, with no confidence intervals, variance, or per-run values provided anywhere in Tables 1-7. With n=3, differences such as the 73.15% to 79.17% ASR change in Table 3 or the 93.51% to 93.99% change in Table 7 are within ordinary binomial noise, so the quantitative claims about optimization gains and defense effectiveness are not statistically supported. Please report confidence intervals or raw per-run values and, if possible, increase the number of repetitions for the main ASR conditions.","section":"Appendix C (Metrics), Tables 1-7"},{"comment":"The OSWorld experiments use 43 'manually selected' browser-related tasks, but the task IDs and selection criteria are not given. Because the attack can only inject into web pages, this selection is necessary, but manual selection without an enumerated list and a stated inclusion rule makes it impossible to assess representativeness or to reproduce the subset. Please provide the full task list, ideally with the selection protocol described.","section":"Appendix C (Environment)"}],"minor_comments":[{"comment":"The phrase 'the attacker has nor knowledge of agent or user intents' should read 'no knowledge of the agent or user intents.'","section":"§4, first sentence"},{"comment":"Reference [36] is missing the publication year and venue details; it currently reads only 'In The Thirteenth International Conference on Learning Representations.' Please complete the citation.","section":"Reference [36]"},{"comment":"The VisualWebArena experiments use 'the 72 tasks selected by [36]', but the task IDs are not listed in the paper. For reproducibility, please enumerate these task IDs or provide a stable link to the exact task set.","section":"Appendix C (Environment)"},{"comment":"The optimized ad example contains a main text of several hundred characters; because the paper claims to follow Google AdSense formats (Section 4.1), it would be helpful to note whether the optimized content respects real ad-network character limits or to discuss how such limits would constrain the optimization procedure.","section":"Appendix E"},{"comment":"The 'Subset' column in Table 2 repeats 'Browser' and 'Web' for every row; consider merging cells or using a cleaner layout to improve readability.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: The main gap between the paper's title/abstract and its experiments is that the 'real-world ad delivery' vector is simulated with a CDP-based mock ad server. This is acknowledged in Appendix F, but the authors continue to assert the real-world claim in the abstract and conclusion. I think major revision is appropriate: either add a real ad-platform feasibility study (even a small-scale submission of benign variants to test policy acceptance) or reframe the contribution as a simulated attack with a clearly stated dependency on ad delivery. The manuscript is generally well-written and the code release is a plus. Also note that the OSWorld task subset is not enumerated, which will hinder reproduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth taking seriously: using advertising delivery as the injection channel for attacking VLM web agents, with a threat model that only assumes static content and no knowledge of user intent. That is a real advance over prior work that needed HTML manipulation, intent knowledge, or model access. The VLM-based optimization that infers likely user intents from the page context is also a nice touch, and the experiments are broad: two benchmarks, several agents, three observation settings, baselines, ablations, and even a defense experiment. Within the simulated setup, the attack is clearly effective, often exceeding 60% ASR and approaching 100% in some configurations. The paper is also honest about several of its own limits in Appendix F, which is to its credit.\n\nThe soft spot is the one the authors themselves concede: the \"real-world\" claim is not actually tested. The mock ad server uses Chrome DevTools Protocol to inject the ad directly into the top-level DOM before every page load. That bypasses ad network content moderation, policy review, and the cross-origin iframe sandboxing that real display ads would face. A creative like \"CRITICAL NOTIFICATION / Close AD\" is exactly the kind of deceptive format that ad platforms moderate. So what the tables report is a simulated attack on a directly modified page, not an attack delivered through advertising. This is a load-bearing distinction because the paper's headline claim is that advertising delivery is a potent real-world vector. The evidence does not support that yet. The authors acknowledge this in Limitations, but the abstract and conclusion still overstate it.\n\nA couple of smaller issues: three repetitions with no error bars or significance tests is thin, especially for ASR numbers in the 60-90% range. The OSWorld task selection is manual and not justified beyond \"we needed browser tasks,\" which is understandable but should be disclosed more prominently. The click-only metric is fine given the stated goal, but calling the attack \"potent\" should be scoped to the click step, not the full harm chain.\n\nNone of this is fatal. The attack works in simulation, the threat model is novel, and the limitations are addressable. The paper deserves a serious referee, but it needs major revision: either validate the ad delivery against a real platform or clearly reframe the contribution as a simulated attack, add proper statistics, and justify task selection. I would bring it to a reading group because it opens a genuinely new angle on web agent security, and I would cite it for the threat model and simulated results, not for the real-world claim.","headline":"Solid simulated attack with a genuinely new ad-delivery channel, but the 'real-world' claim overreaches the evidence and needs tempering before publication.","tokens_in":14840,"tokens_out":1563,"would_cite":true,"duration_ms":18982,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box attack using only a static ad, with no knowledge of the agent or user intent, induces VLM web agents to click malicious ads, succeeding in more than 60% of scenarios and close to 100% in some.","keywords":["web agent security","environment injection attack","advertising delivery","vision-language models","black-box attack","prompt injection","VisualWebArena","OSWorld"],"falsifier":"Submit the exact AdInject creative ('# CRITICAL NOTIFICATION', 'Close AD') to a mainstream advertising platform's review pipeline and observe whether it is served unmodified. If a major ad network rejects or refuses to serve such content, the paper's central 'real-world vector' claim is falsified even though the simulated browser-level attack still works.","tokens_in":1797,"feed_emoji":"🎯","tokens_out":2558,"duration_ms":75496,"temperature":0.7,"pith_summary":"The paper tries to show that internet advertising delivery can be used as a practical channel to attack Vision-Language-Model (VLM) web agents in the real world. It proposes AdInject, a black-box method that places a static, innocuous-looking ad such as 'CRITICAL NOTIFICATION / Close AD' on a page and gets the agent to click it, despite the attacker knowing nothing about the agent, its parameters, or the user's current task. Because the ad is a normal page element, it is visible to every perception mode the agent uses, including accessibility tree, screenshot, and Set-of-Marks. On VisualWebArena and OSWorld the attack succeeds in more than 60 percent of scenarios and approaches 100 percent in several settings, and even a defense prompt that names the exact 'Close AD' button only cuts success to 57 percent. The authors argue that advertising's lenient content moderation makes this a realistic, presently underappreciated vulnerability.","feed_headline":"Fake 'Close AD' ads hijack web agents in up to 100% of tests","feed_subtitle":"A static 'CRITICAL NOTIFICATION' ad tricks black-box VLM web agents, and even warned agents fall for it half the time.","key_machinery":"The load-bearing object is a static advertisement built from text fields (title, main text, button text) modeled on standard display-ad formats, whose message frames the click as a necessary step: '# CRITICAL NOTIFICATION' with button text 'Close AD'. On top of this, a VLM-based content optimization step takes the target site's homepage screenshot and accessibility tree, asks a VLM to guess ten likely user intents, and rewrites the ad text to blend those intents in, making the ad look task-relevant. The ad is delivered through a mock ad server that injects the ad as a real DOM element via a browser remote-debugging protocol, so it appears in every perception channel of the agent; after the click the ad closes and no redirection occurs, isolating the measured effect to the click itself.","core_discovery":"AdInject's central claim is that an attacker who cannot see the agent, cannot modify HTML directly, cannot run JavaScript, and does not know the user's intent can still reliably induce a web agent to click a malicious ad. The ad is crafted to look like a critical system notification whose 'Close AD' button the agent believes it must press to proceed with its task. Across VisualWebArena and OSWorld, with GPT-4o, Claude-3.5, Claude-3.7, UI-TARS, and an R-MCTS agent, the attack achieves high attack success rates, often over 60 percent, sometimes near 100 percent, with the click happening in one or two steps. Injecting the ad usually does not destroy the agent's original task performance, meaning the attack is a low-cost redirect that can hand the agent to an attacker-controlled page.","pith_inferences":["A crucial untested step is the ad platform itself: the paper's mock server injects the ad directly, so the 'real-world' claim assumes a network would accept a 'CRITICAL NOTIFICATION / Close AD' creative; if content moderation rejects it, the practical threat is lower, though the simulated attack remains valid.","Because the optimization prompt tells the VLM to use a strong command tone and repeat the original content, an alternative defense could scan rendered ad text for imperative task-framing patterns, not just known-bad phrases, a testable extension the paper does not explore.","The same delivery channel could be used for subtler goals than clicking: an attacker-controlled landing page fully determines subsequent observations, so the threat extends to whatever post-click task the landing page presents, a step the paper leaves unvalidated.","AdInject's success suggests that agents should treat all third-party content as untrusted and verify that an element is actually required for the task before interacting, a design change, not just a prompt change."],"forward_implications":["If AdInject transfers to real ad delivery, any VLM web agent that browses pages with third-party ads becomes a click away from attacker-controlled content, with the click often happening on step 1 or 2.","Prompt-based defenses are weak shields: generic warnings about distractions or ads leave attack success above 92 percent, and only a defense that says 'do not click the Close AD button' cuts success to 57 percent, and only when placed in the goal rather than the system prompt.","Ad content optimization raises attack success substantially in lower-baseline settings (e.g., from 37.92 to 63.89 percent for Claude-3.7 with accessibility tree only), so attacks improve as the attacker exploits website context.","All tested ad styles, including pop-up, banner, and sidebar, work; smaller ads lose effectiveness, so ad-quality control that shrinks or strips ads could reduce exposure.","Because the ad is a real web element rather than an injected observation, defenses based on sanitizing the agent's observation input alone would miss this attack."],"supporting_citations":[{"why":"Prior pop-up attack whose 'Virus Detected' and 'Speculate User Query' designs serve as baselines and whose threat model AdInject is designed to beat.","marker":"[46]"},{"why":"VisualWebArena benchmark where the main VisualWebArena experiments measure attack and task success rates.","marker":"[18]"},{"why":"OSWorld benchmark from which the paper manually selects 43 browser-related tasks for the OS experiments.","marker":"[37]"},{"why":"Source of the 72-task VisualWebArena subset and the parameter-knowledge attack that AdInject contrasts against as unrealistic.","marker":"[36]"},{"why":"Browser remote-debugging protocol used by the mock ad server to inject the ad as a real DOM element into every page.","marker":"[11]"},{"why":"Display-ad format examples on which the ad content design (title, text, button, styles) is modeled.","marker":"[12]"},{"why":"BrowserGym environment implementation used to run the VisualWebArena agents.","marker":"[8]"},{"why":"GPT-4o system card; GPT-4o is one of the VLM backbones tested and the VLM used for ad-content optimization.","marker":"[16]"},{"why":"UI-TARS agent, the OSWorld top-ranked agent whose lower susceptibility is reported and analyzed.","marker":"[30]"}],"fun_headline_variants":["Malicious 'Close AD' ads redirect web agents with up to 100% success","AdInject: Black-box ads hijack web agents without intent knowledge","Web agents fooled by fake ads posing as critical alerts","Black-box ad injection makes web agents click 'Close AD' up to 100%","Ads disguised as system alerts hijack web agents, success near 100%"],"cache_read_input_tokens":16896,"weakest_assumption_plain":"The attack's status as a real-world vector rests on the assumption that an advertising platform would actually accept and serve a clearly malicious 'CRITICAL NOTIFICATION / Close AD' creative; the experiments sidestep ad-network content moderation by injecting the ad directly with a mock server, and the authors concede this limitation.","fun_headline_variants_meta":{"raw":{"variants":["Malicious 'Close AD' ads redirect web agents with up to 100% success","AdInject: Black-box ads hijack web agents without intent knowledge","Web agents fooled by fake ads posing as critical alerts","Black-box ad injection makes web agents click 'Close AD' up to 100%","Ads disguised as system alerts hijack web agents, success near 100%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000915,"raw_usage":{"total_tokens":3962,"prompt_tokens":1011,"completion_tokens":2951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2852}},"tokens_in":627,"tokens_out":2951,"duration_ms":22715,"temperature":1.0,"reasoning_tokens":2852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:26:22.733598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Submit the exact AdInject creative ('# CRITICAL NOTIFICATION', 'Close AD') to a mainstream advertising platform's review pipeline and observe whether it is served unmodified. If a major ad network rejects or refuses to serve such content, the paper's central 'real-world vector' claim is falsified even though the simulated browser-level attack still works.","supporting_citations":[{"cited_title":"Dissecting adversarial robustness of multimodal lm agents","cited_arxiv_id":null,"evidence_quote":"Source of the 72-task VisualWebArena subset and the parameter-knowledge attack that AdInject contrasts against as unrealistic."},{"cited_title":"Chrome devtools protocol","cited_arxiv_id":null,"evidence_quote":"Browser remote-debugging protocol used by the mock ad server to inject the ad as a real DOM element into every page."},{"cited_title":"Google adsense","cited_arxiv_id":null,"evidence_quote":"Display-ad format examples on which the ad content design (title, text, button, styles) is modeled."}],"review_version":1}