{"id":"172d36d4-ac6e-468f-ae75-6656ed05ba12","arxiv_id":"2608.04741","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LoginTrap induces LLM-based web agents to enter attacker-controlled login flows and submit sensitive information, achieving 86% average end-to-end success across LLM backbones.","lead":"Researchers show a new attack called LoginTrap that tricks AI web agents into logging into fake login pages and handing over private information. The attack works without knowing the user's task or the agent's design, succeeding in 86% of tests across different AI models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The login form used in every evaluation is rendered with a fixed line-separated layout cited from SpatialJB, a known LLM guardrail jailbreak; without ablating this layout, the reported 86% ASR may measure layout-based compliance rather than login-context inducement.","rationale":"The paper addresses an important and timely question, and the reported consistency across backbones and agent architectures is suggestive that web agents can be manipulated by webpage content. However, the central claim is quantitative: LoginTrap reaches 86% end-to-end ASR, and the evaluation's success condition is defined by the agent entering synthetic sensitive information from the task background into the controlled login form. The login form is described only as using a fixed line-separated layout, with a citation to SpatialJB, a paper whose stated purpose is to use text layout as a jailbreak key against LLM guardrails. That makes the layout a plausible, unstated active ingredient in the attack. If the layout rather than the login context is what drives form completion, the headline result is not evidence for the paper's login-boundary thesis. The reader's realism and reproducibility concerns are valid but less pointed; the SpatialJB confound identifies a concrete, testable dependency in the main result. The paper has no released artifacts, error bars, or defense section despite promising defense experiments, so the evidence is not yet sufficient to accept the claim as stated. A conditional verdict with a requirement for the ablation is the right posture, which matches the existing verdict; hence no change.","tokens_in":15702,"tokens_out":8494,"duration_ms":106196,"concrete_test":"Re-run the RQ1 evaluation on the same 80-page, 1,175-task dataset while replacing only the login page rendering with a standard HTML login form (normal labeled fields, no SpatialJB line-separated layout), keeping the injected popups and all other settings fixed. If ASR drops substantially below the reported 86%, the line-separated jailbreak layout is a necessary component of the attack; if ASR stays near 86%, the layout is not the driver. As a complement, also run a non-login 'profile update' form with the same line-separated layout to test whether the layout alone elicits sensitive-information submission.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that \"The login form requests sensitive fields using a fixed line-separated layout [29]\". Reference [29] is SpatialJB: \"How text distribution art becomes the 'jailbreak key' for LLM guardrails\". This layout is the only description of the login form, and the paper never ablates it. The central claim is that webpage context alone induces agents to cross a login boundary and submit sensitive information, but the evaluation's success condition is defined as entering the synthetic sensitive information from the task background into this controlled form. If the line-separated layout is itself an adversarial presentation that disables LLM guardrails, then a high ASR could be produced by the layout rather than by the login framing or by the page-specific fuzzed injection. The paper does not report results for a conventional login form, nor does it compare against a non-login data-collection form with the same layout, so the contribution of the login context to the measured attack success is unidentified. This is a load-bearing confound: it affects the main quantitative result, not just the realism of the environment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LoginTrap, a task-agnostic indirect prompt injection attack that induces LLM-based web agents to initiate an attacker-controlled login flow and submit sensitive background information. The attacker is assumed to control only the cloned webpage content and the induced login page, without knowing the user task or the agent's internals. The attack generates page-specific injection statements through a fuzzing-inspired process that uses GPT-4o for candidate generation and shadow-LLM evaluation. The evaluation is conducted in a local, controlled environment on 80 cloned pages with 1,175 tasks drawn from Mind2Web, and reports Login Entry Rate, Attack Success Rate, and Page Exploitability. The main results are an 86% average end-to-end attack success across four LLM backbones, 79% across three web agent architectures, and 100% page exploitability within a 9-attempt budget for the tested pages.","tokens_in":15884,"tokens_out":4936,"duration_ms":50916,"significance":"If the central claim holds, LoginTrap would establish that black-box webpage control alone can systematically induce LLM-based web agents to cross the login boundary and leak sensitive information, a distinct and realistic threat model for delegated browsing. The paper's strengths include a conservatively stated threat model (no knowledge of the task, model, or architecture), a task-agnostic generation procedure, and an evaluation spanning multiple backbones and architectures. The proposed page-level exploitability metric with attempt-consumption statistics is a useful complement to task-level ASR. However, the significance is contingent on resolving the login-form layout confound and on reporting the missing defense evaluation, both of which currently leave the main quantitative claims ambiguous.","major_comments":[{"comment":"The login form uses a fixed line-separated layout cited to SpatialJB [29], a known technique for bypassing LLM guardrails. The paper does not ablate the layout, nor does it compare against a conventional login form or a non-login data-collection form with the same layout. As a result, the reported 86% ASR in RQ1 cannot be attributed to the login context or the page-specific injection; it may be produced by the adverarial layout alone. Please add an ablation that varies the form layout while holding the injection and login context fixed, and a control condition with the same layout on a non-login form, to isolate the contribution of login contextualization.","section":"§3.3"},{"comment":"The abstract and conclusion claim that LoginTrap 'remains effective across agent architectures and defenses,' and both §4.1 and §6 mention 'defense settings,' yet no defense experiment is described or reported anywhere in the evaluation. The paper only presents RQ1 and RQ2; there is no table or figure measuring attack success under any defense. The robustness-against-defenses claim is therefore unsupported by the presented evidence and should either be substantiated with new experiments or removed from the abstract and conclusion.","section":"Abstract and §§4–6"},{"comment":"The injection generator uses GPT-4o for candidate generation and shadow-LLM evaluation, and RQ1 evaluates GPT-4o as one of the agent backbones. This creates a same-family selection bias: the accepted candidate set is tuned to be persuasive to GPT-4o-family models, so the GPT-4o row in Figure 3 partly measures the generator's optimization target rather than the attack's generality. Please report whether the results change when the generator uses a different model family, or provide a per-backbone breakdown of accepted versus rejected candidates to quantify this bias.","section":"§4.1 (Implementation)"},{"comment":"The paper does not release the list of cloned webpages, the generated injection statements, the login templates, or the evaluation harness. Given that the environment is fully controlled and the central claims depend on page-level exploitability, these artifacts are necessary for independent verification. Without them, the reader cannot assess whether the 80 pages are representative or whether 'the rest of the webpage is kept unchanged' (Section 3.3) actually holds. Please make these artifacts available or provide a sufficiently detailed specification for re-execution.","section":"§4.1 (Dataset and Artifacts)"}],"minor_comments":[{"comment":"The phrase 'follow key research questions' should be 'the following key research questions.'","section":"§4.1"},{"comment":"The sentence 'requires more more attempts' should be 'requires more attempts.'","section":"§4.2"},{"comment":"The sentence 'complete the complete the controlled login form' contains a duplicated phrase and should be corrected.","section":"§4.3"},{"comment":"The paper uses 'LOGINTRAP' in some figure captions and 'LoginTrap' in the text; please unify the casing consistently.","section":"Figures 3 and 5"},{"comment":"The MCTS-inspired mutation policy is described only qualitatively; please specify the number of mutation rounds, the generation budget, and the reward function used for seed selection so the procedure is reproducible.","section":"§3.4"},{"comment":"The reference to SpatialJB [29] as 'a fixed line-separated layout' is too vague; please state explicitly that this is an adversarial text-layout technique known to bypass LLM guardrails and discuss its role in the attack design.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The login-form layout confound is the key issue: the fixed line-separated layout is a published LLM guardrail jailbreak, and without an ablation the headline ASR numbers may not support the paper's stated claim about login-context inducement. The missing defense evaluation is also a clear mismatch between the abstract's claims and the presented experiments. Both are fixable with additional experiments, so I do not recommend rejection, but the current version is not yet acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LoginTrap is a useful attack formulation: it names login inducement as a distinct failure mode for LLM web agents and gives a clean black-box threat model. The fuzzing-inspired generation pipeline is a real engineering contribution, and splitting LER from ASR is the right analytical move. The page-level PER result—every tested page exploitable within nine attempts—is more meaningful for practical risk than the aggregate ASR, and I believe it.\n\nThe soft spots are mostly about evidence, not framing. The biggest is in Section 3.3: the login form uses a fixed line-separated layout cited to SpatialJB, a known guardrail jailbreak. The paper never ablates a conventional login form. So the 86% end-to-end ASR may partly be measuring layout-driven compliance after the agent reaches the login page, not the login-context inducement that is the paper's actual claim. LER is not confounded by this, and 93% LER alone shows the login-entry risk, but the headline privacy-leakage number needs an ablation before I'd trust it.\n\nSecond, the abstract and conclusion claim effectiveness \"across defenses\" and say several defense settings \"reduce but do not eliminate\" leakage, but Section 4 contains no defense experiments at all. That claim is unsupported in this version and should be removed or the data added.\n\nThird, there are no artifacts: no clone list, no generated injection statements, no evaluation harness. For a prompt-engineering attack, the actual statements and pages are the evidence. That should be a release condition, not a courtesy.\n\nTwo smaller points. The generator and evaluator use GPT-4o in-house and then measure GPT-4o-based agents in the same family; that mild same-family bias should be addressed with a cross-generator condition. And the paper aggregates 1,175 task instances without confidence intervals; a \"log in to continue\" static template baseline would also help calibrate what the fuzzing actually buys.\n\nOn balance, the central phenomenon—web agents can be induced to cross the login boundary through page context alone—is plausible and partly supported. The strength of the effect is not yet established, and the defense claims are unsubstantiated, but this is not a fabricated or incoherent paper. It deserves a serious referee, and the review should ask for the layout ablation, the artifacts, and the defense data. I would not cite the 86% number in its current form.","headline":"A genuinely useful attack formulation with a clean threat model, but the headline ASR is not yet trustworthy until the SpatialJB layout confound is ablated and the missing defense and artifact material appears.","tokens_in":16451,"tokens_out":3982,"would_cite":false,"duration_ms":49439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LoginTrap demonstrates that a black-box attacker controlling only page content can induce LLM web agents to enter fake login flows and submit sensitive information, with 86% average end-to-end success across LLM backbones and universal…","keywords":["indirect prompt injection","web agents","login inducement","phishing","authentication boundary","privacy leakage","task-agnostic attack","LLM security"],"falsifier":"Run the generated LoginTrap pages as live sites with real authentication forms and real (sandboxed) credentials, or add a browser-level phishing warning overlay before the login page; if agents stop submitting sensitive information in either condition, the attack relies on the controlled local environment rather than on general webpage control.","tokens_in":15465,"feed_emoji":"🎣","tokens_out":6713,"duration_ms":67761,"temperature":0.7,"pith_summary":"This paper tries to establish that login inducement is a systematic, task-agnostic attack surface for LLM web agents: an attacker who controls only what appears on a webpage can make an agent treat login as a necessary prerequisite, enter a fake login flow, and submit sensitive information, all without knowing the user's task or the agent's internals. The proposed attack, LoginTrap, generates page-specific injection text with a fuzzing-inspired process and embeds it as a popup on a cloned benign page. Across 80 webpages, 1,175 tasks, five website domains, four LLM backbones, and three agent architectures, the attack reaches an average 86% end-to-end success across backbones and 79% across architectures, with every tested page exploitable within nine attempts. A sympathetic reader would care because this turns the routine login prompt into a privacy-leakage channel that current defenses do not fully close.","feed_headline":"Fake login prompts make AI web agents leak data 86% of the time","feed_subtitle":"Task-agnostic attack works across models and agent designs; every tested page was exploitable within nine attempts.","key_machinery":"The carrying mechanism is the fuzzing-inspired injection generator combined with a controlled login surface. The generator extracts a compact summary and a benign probe task from each page, initializes candidate statements from semantic strategies that frame login as relevant to the page, filters them with two shadow LLMs (one with an extra reminder to be suspicious of webpage instructions), and when none pass, mutates failed candidates using a Monte-Carlo-tree-search-inspired seed-selection policy with crossover, expansion, rephrasing, and compression. The accepted statement is placed in a popup container on a clone of the original page, with the rest of the page unchanged, and clicking the forged login entry leads to an attacker-controlled login page whose description is LLM-generated from the page summary and whose sensitive fields use a fixed line-separated layout. This makes login appear as a plausible, page-consistent prerequisite without directly asking the agent to leak information.","core_discovery":"The paper's central claim is that a task-agnostic, black-box attacker who controls only the webpage can induce an LLM web agent to treat login as a necessary step and submit sensitive information into an attacker-controlled form. LoginTrap instantiates this by cloning a benign page, inserting a popup with a page-specific login-inducing statement, and routing the agent to a controlled login page. In the evaluation, the average login entry rate was 93% and the average end-to-end attack success was 86% across four LLM backbones with one agent framework; with one fixed LLM and three agent architectures, the average attack success was 79%, and within a nine-attempt budget every tested webpage in every domain was exploitable under at least one backbone.","pith_inferences":["Editorial inference: agents that are better at completing tasks once they accept a premise may become more vulnerable, so future improvements in task-following could amplify login-inducement attacks rather than fix them.","Editorial inference: the controlled environment's five-step budget and cloned pages mean real-world transfer is still open; live deployments with password-manager prompts, phishing warnings, or richer page states could lower success rates.","Editorial inference: a practical defense could gate any login or sensitive-form action on whether authentication is needed for the stated user task, which is directly testable with the paper's dataset."],"forward_implications":["An attacker who can place content on any page the agent visits, such as through ads or compromised pages, can push the agent into a login flow and collect sensitive information even when the user's task has nothing to do with login.","Swapping the underlying LLM or the agent architecture does not remove the risk; every tested backbone and architecture could be induced to enter the forged login path.","Because every tested webpage became exploitable within nine attempts, the risk is page-level and systematic, not limited to a few vulnerable sites.","Existing defenses at the instruction, system-prompt, and action-supervision layers reduce leakage but do not eliminate it, so defenses must explicitly reason about whether login is necessary for the task."],"supporting_citations":[{"why":"Establishes that login pages are common across the web, motivating login as a realistic attack boundary.","marker":"[1]"},{"why":"Supplies the Monte-Carlo-tree-search-inspired seed selection used to mutate failed injection candidates.","marker":"[7]"},{"why":"Provides the real-world webpage and user task dataset from which the evaluation pages and tasks are drawn.","marker":"[10]"},{"why":"Systematizes prior web-agent prompt-injection benchmarking that this work extends by targeting the login boundary.","marker":"[13]"},{"why":"Provides the line-separated text layout that lets the controlled login form present sensitive fields in a parseable, fixed format.","marker":"[29]"},{"why":"The direct tool-loop web agent used as the fixed architecture in the cross-backbone experiments and as one architecture in the cross-agent experiments.","marker":"[30]"},{"why":"Supplies the multi-agent workflow architecture tested in the cross-architecture generalization experiments.","marker":"[39]"},{"why":"Supplies the modular, separate-grounding web agent architecture tested in the cross-architecture generalization experiments.","marker":"[49]"}],"fun_headline_variants":["Task-agnostic attack makes AI web agents hand over credentials 86% of the time","LoginTrap: 86% success inducing AI web agents to leak credentials","Webpage-only attacker tricks LLM agents into login, leaking data 86%","Black-box login-inducing attack: 86% end-to-end success on web agents","Every page cracked in under 9 tries: web agents tricked into login"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a locally cloned page with an injected popup and a template-generated login form is a faithful proxy for a real website with genuine authentication, so agents will act the same way on live services.","fun_headline_variants_meta":{"raw":{"variants":["Task-agnostic attack makes AI web agents hand over credentials 86% of the time","LoginTrap: 86% success inducing AI web agents to leak credentials","Webpage-only attacker tricks LLM agents into login, leaking data 86%","Black-box login-inducing attack: 86% end-to-end success on web agents","Every page cracked in under 9 tries: web agents tricked into login"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3494,"prompt_tokens":933,"completion_tokens":2561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":549,"tokens_out":2561,"duration_ms":17980,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:36:03.976737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the generated LoginTrap pages as live sites with real authentication forms and real (sandboxed) credentials, or add a browser-level phishing warning overlay before the login page; if agents stop submitting sensitive information in either condition, the attack relies on the controlled local environment rather than on general webpage control.","supporting_citations":[{"cited_title":"A {Large-Scale} measurement of website login policies","cited_arxiv_id":null,"evidence_quote":"Establishes that login pages are common across the web, motivating login as a realistic attack boundary."},{"cited_title":"Efficient selectivity and backup operators in monte-carlo tree search","cited_arxiv_id":null,"evidence_quote":"Supplies the Monte-Carlo-tree-search-inspired seed selection used to mutate failed injection candidates."},{"cited_title":"Mind2web: Towards a generalist agent for the web","cited_arxiv_id":null,"evidence_quote":"Provides the real-world webpage and user task dataset from which the evaluation pages and tasks are drawn."},{"cited_title":"Wasp: Benchmarking web agent security against prompt injection attacks.Advances in Neural Information Processing Systems 38(2026)","cited_arxiv_id":null,"evidence_quote":"Systematizes prior web-agent prompt-injection benchmarking that this work extends by targeting the login boundary."},{"cited_title":"jailbreak key","cited_arxiv_id":"2601.09321","evidence_quote":"Provides the line-separated text layout that lets the controlled login form present sensitive fields in a parseable, fixed format."},{"cited_title":"Browser use: Enable ai to control your browser","cited_arxiv_id":null,"evidence_quote":"The direct tool-loop web agent used as the fixed architecture in the cross-backbone experiments and as one architecture in the cross-agent experiments."},{"cited_title":"Skyvern: Automate browser-based workflows with ai","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-agent workflow architecture tested in the cross-architecture generalization experiments."},{"cited_title":"Litewebagent: The open-source suite for vlm-based web-agent applications","cited_arxiv_id":null,"evidence_quote":"Supplies the modular, separate-grounding web agent architecture tested in the cross-architecture generalization experiments."}],"review_version":1}