{"id":"e0a8f3d1-5707-4abb-ad10-fb5d8a91dc1c","arxiv_id":"2605.15030","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"WARD is a guard model trained on 177K web samples and adversarially hardened via attacker-guard co-evolution to achieve high recall on prompt injections with low false positives and no added latency.","lead":"The paper introduces WARD, a guard model for web agents using large-scale datasets from real websites and an adaptive co-evolution training method called A3T to defend against prompt injection attacks. A smart generalist might read it to learn about practical security measures for AI systems that interact with the open web.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Dataset coverage assumption for OOD generalization and robustness claims is unverified","rationale":"The reader's weakest_assumption directly identifies the dataset coverage issue as load-bearing for all performance claims. This matches the abstract's reliance on those datasets for OOD and robustness assertions. No internal inconsistency or other technical flaw is detectable from available text; the primary barrier remains lack of full manuscript details for further verification.","tokens_in":1735,"tokens_out":319,"duration_ms":19890,"concrete_test":"Sample 100 new URLs outside the original 719 (e.g., from a disjoint slice of top-traffic lists or non-English domains), generate fresh prompt-injection variants not derived from WARD-PIG, and re-evaluate WARD recall/FP/robustness; if recall falls below 95% or robustness degrades, the generalization claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims—nearly perfect recall on out-of-distribution benchmarks, low false positives, and robustness to guard-targeted/adaptive attacks under distribution shifts—rest on WARD-Base (177K samples from 719 high-traffic URLs) and WARD-PIG providing sufficient coverage of real-world web content and evolving attacks. No details establish that the 719 URLs span domain diversity, site structures, or languages, nor that WARD-PIG attacks (even with A3T co-evolution) represent threats outside the training loop. If coverage is narrow, reported OOD performance and robustness may reflect in-distribution behavior rather than true generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes WARD, a guard model for defending web agents against prompt injection attacks in HTML and visual interfaces. It is constructed from the WARD-Base dataset (~177K samples collected from 719 high-traffic URLs and platforms) and the WARD-PIG dataset for guard-targeted attacks, trained via the A3T framework that performs iterative co-evolution between a memory-based attacker and the guard. The central experimental claims are nearly perfect recall on out-of-distribution benchmarks, low false-positive rates that preserve agent utility, robustness to both guard-targeted and adaptive attacks under distribution shifts, and efficient parallel execution with no added latency.","tokens_in":1847,"tokens_out":562,"duration_ms":17514,"significance":"If the reported results hold under rigorous verification, this would be a meaningful practical contribution to securing autonomous web agents, directly addressing the four stated limitations of prior guards (generalization, false positives, latency, and adversarial vulnerability). The scale of the collected data and the explicit co-evolutionary training loop are strengths that could influence future defense design.","major_comments":[{"comment":"§3 (Dataset construction): The OOD generalization and robustness claims rest on the assumption that the 719 high-traffic URLs in WARD-Base provide sufficient domain, structural, and linguistic diversity to support performance under 'substantial distribution shifts.' No quantitative analysis (e.g., domain entropy, language distribution, or structural feature coverage) is supplied to substantiate this coverage, leaving open the possibility that reported OOD results reflect in-distribution behavior rather than true generalization.","section":"§3"},{"comment":"§4.3 and §5 (Evaluation and A3T): The robustness claims against adaptive and guard-targeted attacks are supported by A3T co-evolution, yet the paper does not report whether the held-out OOD benchmarks and WARD-PIG test splits were generated independently of the co-evolution loop or whether any post-hoc selection of attack variants occurred. This detail is load-bearing for the claim that robustness extends 'outside the training loop.'","section":"§4.3, §5"}],"minor_comments":[{"comment":"The abstract states 'nearly perfect recall' and 'low false positive rates' without numerical values or confidence intervals; these should be stated explicitly with the corresponding tables or figures in the main text.","section":"Abstract"},{"comment":"Notation for the memory-based attacker components in the A3T description could be clarified with a small diagram or pseudocode to improve reproducibility.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and indicate planned revisions.","responses":[{"response":"We agree that quantitative diversity metrics would strengthen the justification for OOD claims. The 719 URLs were selected from high-traffic sources across multiple platforms to capture varied web structures, but explicit entropy or coverage statistics were omitted. In revision we will add domain, language, and structural feature analysis to §3.","revision_made":"yes","referee_comment":"[§3] §3 (Dataset construction): The OOD generalization and robustness claims rest on the assumption that the 719 high-traffic URLs in WARD-Base provide sufficient domain, structural, and linguistic diversity to support performance under 'substantial distribution shifts.' No quantitative analysis (e.g., domain entropy, language distribution, or structural feature coverage) is supplied to substantiate this coverage, leaving open the possibility that reported OOD results reflect in-distribution behavior rather than true generalization."},{"response":"The OOD benchmarks and WARD-PIG test splits were constructed and held out independently of the A3T training loop; the co-evolution operated exclusively on training data with no post-hoc variant selection on test sets. We will add explicit statements confirming this separation in the revised §4.3 and §5.","revision_made":"yes","referee_comment":"[§4.3, §5] §4.3 and §5 (Evaluation and A3T): The robustness claims against adaptive and guard-targeted attacks are supported by A3T co-evolution, yet the paper does not report whether the held-out OOD benchmarks and WARD-PIG test splits were generated independently of the co-evolution loop or whether any post-hoc selection of attack variants occurred. This detail is load-bearing for the claim that robustness extends 'outside the training loop.'"}],"tokens_in":1456,"tokens_out":409,"duration_ms":19760,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main point is a guard model called WARD that uses a new 177K-sample dataset from 719 high-traffic URLs plus a dedicated attack set, trained via an iterative memory-based attacker-guard loop called A3T. The abstract claims near-perfect OOD recall, low false positives, robustness to adaptive and guard-targeted attacks, and no added latency.\n\nWhat is actually new is the scale and source of WARD-Base, the WARD-PIG attack collection, and the co-evolution setup tailored to web agents. The focus on running in parallel with the agent and handling evolving threats is a direct response to known weaknesses in existing guards.\n\nThe approach is reasonable on paper for a practical security tool. Collecting from real high-traffic sites is better than pure synthetic data, and the iterative training tries to close the loop on adaptive attacks.\n\nThe soft spots are exactly where the stress-test note flags them. The abstract gives zero information on data splits, baselines, how OOD is defined, or how attacks outside the training loop were constructed. Without those, the generalization and robustness numbers cannot be evaluated. The 719 URLs may be too narrow in domain or structure to support the claimed coverage of real-world shifts. That is a load-bearing gap, not a minor one.\n\nThis is for researchers working on securing autonomous web agents. Someone building or evaluating defenses in that area could use the datasets if released.\n\nIt deserves peer review because the problem is timely and the method is concrete enough to test once the experiments are shown. Send it out so the claims can be checked against the actual results and setup.","headline":"New datasets and A3T co-evolution loop for web agent prompt injection defense, but no experimental details to check the robustness claims.","tokens_in":2336,"tokens_out":405,"would_cite":false,"duration_ms":25998,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"WARD is a guard model that defends web agents from prompt injections with near-perfect recall on unseen data, low false positives, and no added latency while resisting adaptive attacks.","keywords":["web agents","prompt injection","adversarial defense","guard model","adversarial training","web security","AI agents","prompt injection attacks"],"falsifier":"Running WARD on a fresh collection of web pages and prompt-injection examples gathered after the original datasets and measuring whether recall falls below 95 percent or false-positive rate rises above 5 percent on those new examples.","tokens_in":2645,"feed_emoji":"🛡️","tokens_out":647,"duration_ms":19396,"temperature":0.7,"pith_summary":"Web agents that browse and act on websites are open to prompt injection attacks hidden in page content. The paper presents WARD as a dedicated guard built on a 177K-sample dataset drawn from 719 real high-traffic sites plus a second dataset focused on attacks against the guard itself. It adds an iterative training loop in which an attacker and the guard improve against each other. The resulting model is claimed to generalize to new domains, avoid blocking normal pages, stay effective when attackers target the guard directly, and run alongside the agent without slowing it down. If these properties hold, web agents could be used more safely in open environments.","feed_headline":"WARD guard blocks prompt injections in web agents with near-perfect recall","feed_subtitle":"Trained on large real-web data and adaptive attacker loops, it runs in parallel without adding latency or raising false blocks.","key_machinery":"A3T adaptive adversarial attack training framework that iteratively strengthens the guard through memory-based attacker and guard co-evolution.","core_discovery":"WARD is a guard model trained on WARD-Base (approximately 177K samples from 719 high-traffic URLs) and WARD-PIG (a prompt-injection dataset targeting the guard), using the A3T adaptive adversarial attack training framework that performs memory-based attacker-guard co-evolution; it achieves nearly perfect recall on out-of-distribution benchmarks, low false-positive rates, robustness to guard-targeted and adaptive attacks under distribution shifts, and parallel execution with the agent that adds no latency.","pith_inferences":["The co-evolution training loop could be applied to other language-model agents that face similar injection risks outside web browsing.","Periodic retraining on newly collected pages would likely be required as website structures and attack techniques change.","Production deployment on actual user-facing agents would provide a direct test of whether the reported efficiency and robustness translate to real traffic.","The approach might extend to visual prompt injections in multimodal agents if the guard is updated to process screenshots alongside text."],"forward_implications":["Web agents can complete tasks on live websites with reduced risk of being hijacked by injected instructions.","Agent utility stays intact because benign content is rarely misclassified as malicious.","The defense continues to work when attackers deliberately craft inputs to fool the guard model itself.","No extra waiting time is added at each agent step because the guard runs in parallel.","The same training approach could be reused whenever new attack patterns appear."],"fun_headline_variants":["WARD guards web agents from prompt injections with near-perfect recall","A3T training gives WARD robustness against adaptive prompt injection attacks","WARD runs parallel to agents with low false positives on 177K web samples","WARD-PIG dataset boosts defense to unseen prompt injection patterns"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The datasets collected from high-traffic sites and the dedicated prompt-injection set are assumed to cover enough real-world content and attack variations for the reported generalization and robustness to hold.","fun_headline_variants_meta":{"raw":{"variants":["WARD guards web agents from prompt injections with near-perfect recall","A3T training gives WARD robustness against adaptive prompt injection attacks","WARD runs parallel to agents with low false positives on 177K web samples","WARD-PIG dataset boosts defense to unseen prompt injection patterns"]},"model":"grok-4.3","cost_usd":0.004446,"raw_usage":{"total_tokens":2233,"prompt_tokens":694,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":44462000,"prompt_tokens_details":{"text_tokens":694,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1468,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":694,"tokens_out":71,"duration_ms":15489,"temperature":1.0,"reasoning_tokens":1468,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T20:14:05.256277+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running WARD on a fresh collection of web pages and prompt-injection examples gathered after the original datasets and measuring whether recall falls below 95 percent or false-positive rate rises above 5 percent on those new examples.","supporting_citations":[],"review_version":1}