WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

Asmit Nayak; Basieem Siddique; Guruprasad Viswanathan Ramesh; Kassem Fawaz

arxiv: 2604.06367 · v2 · pith:WD3B2JZE · submitted 2026-04-07 · cs.CR · cs.AI· cs.LG

WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

Guruprasad Viswanathan Ramesh , Asmit Nayak , Basieem Siddique , Kassem Fawaz This is my paper

Reviewed by Pith2026-05-10 18:44 UTCgrok-4.3pith:WD3B2JZE open to challenge →

classification cs.CR cs.AIcs.LG

keywords web agentssecurity and privacy tasksevaluation benchmarkstateful UI elementsmultimodal modelsbrowser automationprivacy settingstask failure analysis

0 comments

The pith

Current web agents fail more than 45 percent of the time on security and privacy tasks that use stateful UI elements such as toggles and checkboxes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper fills a gap by creating WebSP-Eval, a benchmark with 200 hand-crafted task instances spanning 28 websites that tests whether web agents can complete everyday user security and privacy actions such as setting cookie preferences or revoking sessions. It supplies a supporting system that resets account states consistently via a custom browser extension and then runs eight different agents built on multimodal large language models. The results show these agents lack reliable autonomous exploration skills, perform unevenly across task categories and sites, and encounter their highest failure rates on pages containing stateful controls. A sympathetic reader would care because web agents are already being deployed for routine browser work, so their inability to manage privacy settings reliably could expose users to unwanted tracking or data leaks.

Core claim

WebSP-Eval demonstrates that state-of-the-art multimodal agents exhibit limited autonomous exploration when executing website security and privacy tasks, leading to poor performance on specific task categories and websites, with stateful UI elements such as toggles and checkboxes emerging as the dominant failure mode at rates exceeding 45 percent across many models.

What carries the argument

WebSP-Eval framework, consisting of the 200-task dataset, a Chrome extension for consistent account and state initialization, and an automated evaluator; the framework isolates performance drops tied to stateful UI components.

If this is right

Developers of web agents must prioritize better handling of dynamic, state-dependent controls to raise success rates on privacy tasks.
Future benchmarks for web agents should include dedicated security and privacy task suites to expose these weaknesses systematically.
Performance gaps across websites indicate that agent training or prompting needs site-specific adaptation rather than generic approaches.
The state-management extension enables repeatable evaluation, allowing direct comparison of future agent improvements on the same tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If stateful elements are the main bottleneck, training corpora for agents could be enriched with many more examples of checkbox and toggle interactions inside privacy flows.
The observed exploration limits may point to a wider difficulty for agents in maintaining context across multi-step, state-changing web sessions beyond security tasks.
Widespread adoption of such agents without fixes could inadvertently reduce user control over personal data settings on popular sites.

Load-bearing premise

The 200 manually written tasks across 28 websites represent the actual diversity and frequency of real-world user-facing security and privacy interactions, and the custom extension maintains identical starting states without introducing artifacts.

What would settle it

Re-running the same agents on a fresh collection of tasks that deliberately varies the proportion and types of stateful UI elements and websites, then measuring whether the failure rate on those elements drops below 45 percent or stays stable.

Figures

Figures reproduced from arXiv: 2604.06367 by Asmit Nayak, Basieem Siddique, Guruprasad Viswanathan Ramesh, Kassem Fawaz.

**Figure 2.** Figure 2: Modules of the WebSP-Eval evaluation framework: 1) Task Curation – Curation of a dataset consisting of website security and privacy tasks across websites. 2) Agent Instantiation – A novel web agent deployment supporting account and state management, utilizing an MLLM and a Selenium driven backbone to execute actions 3) Automated Verification – An automated Vision Language Model-based judge to assess agent … view at source ↗

**Figure 3.** Figure 3: Failure example highlighting website specific design on Steam (Gemini-3-Pro, [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

**Figure 4.** Figure 4: Success example for Gemma-3-27b on the W/oNav variant. Given the instruction “Disable the notifications for cake day updates.”, the model successfully navigates to the page and clicks on the Cake Day updates option and disables notifications. Step 1: Click [9] Step 2: Click [12] Step 3: Click [18] Step 4: Click [29] Step 5: Answer- Task Solved [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗

**Figure 5.** Figure 5: Success example for Claude-Haiku-4.5 on the [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗

**Figure 6.** Figure 6: Comparison of Gemini-3-Pro trajectories on Twitch with (top) and without (bottom) explicit navigational [PITH_FULL_IMAGE:figures/full_fig_p029_6.png] view at source ↗

**Figure 7.** Figure 7: Failure example highlighting website specific design on Duolingo (Gemini-2.5-Pro, [PITH_FULL_IMAGE:figures/full_fig_p030_7.png] view at source ↗

**Figure 8.** Figure 8: Failure example highlighting a Cookie & Tracking Consent Management task failure on Docker (GPT-5- [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗

read the original abstract

Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate general-purpose performance~(e.g., WebArena) or safety against malicious actions~(e.g., SafeArena), no existing framework assesses an agent's ability to successfully execute user-facing website security and privacy tasks, such as managing cookie preferences, configuring privacy-sensitive account settings, or revoking inactive sessions. To address this gap, we introduce WebSP-Eval, an evaluation framework for measuring web agent performance on website security and privacy tasks. WebSP-Eval comprises 1) a manually crafted task dataset of 200 task instances across 28 websites; 2) a robust agentic system supporting account and initial state management across runs using a custom Google Chrome extension; and 3) an automated evaluator. We evaluate a total of 8 web agent instantiations using state-of-the-art multimodal large language models, conducting a fine-grained analysis across websites, task categories, and UI elements. Our evaluation reveals that current models suffer from limited autonomous exploration capabilities to reliably solve website security and privacy tasks, and struggle with specific task categories and websites. Crucially, we identify stateful UI elements are a primary reason for agent failure, with toggles causing more than 45% task failure across many models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

WebSP-Eval fills a real gap with the first benchmark for web agents on privacy tasks and flags clear failure modes, but the methods section needs more detail to hold up.

read the letter

This paper introduces WebSP-Eval, the first framework to test web agents specifically on user-facing security and privacy tasks like managing cookies or account settings. The central finding is that current models have trouble with autonomous exploration and fail more than 45% of the time when tasks involve stateful UI elements such as toggles and checkboxes. The work does a good job filling the gap between general web benchmarks and safety ones by creating 200 task instances on 28 real websites. They include a custom Chrome extension for managing accounts and resetting states, plus an automated evaluator. Testing eight agent variants from multimodal LLMs and breaking down results by website, task type, and UI element gives useful granularity on where things break. The main concern is that the abstract and available details don't spell out how the tasks were validated or what counts as success in each case. It's also not clear if the 200 tasks represent typical real-world privacy actions or if the state management tool introduces any artifacts. With a manually curated set, selection bias is always a risk, though the stress test didn't flag inconsistencies. Overall, this is for researchers developing web agents who care about privacy and security implications. It shows honest engagement with the literature by identifying the specific gap and providing a reproducible setup. The math and data here are empirical measurements rather than complex models, so the citation pattern looks fine. I would bring this to a reading group for discussion on agent evaluation methods. It deserves peer review because the idea is timely and the evaluation setup is practical, even if revisions will likely strengthen the methods section.

Referee Report

2 major / 2 minor

Summary. The paper introduces WebSP-Eval, a benchmark framework for evaluating web agents on user-facing website security and privacy tasks such as cookie management, privacy settings, and session revocation. It consists of a manually curated dataset of 200 task instances spanning 28 websites, a custom Chrome extension for consistent account and initial-state management, and an automated evaluator. The authors evaluate eight agent instantiations based on state-of-the-art multimodal LLMs, performing fine-grained analysis by website, task category, and UI element type. Key findings include limited autonomous exploration capabilities overall and a failure rate exceeding 45% on tasks involving stateful UI elements such as toggles and checkboxes.

Significance. If the empirical results hold under rigorous validation, the work provides a timely benchmark that highlights a previously under-examined weakness in web agents: reliable handling of interactive, state-dependent security and privacy interfaces. The identification of stateful UI elements as a dominant failure mode offers a concrete, actionable direction for agent improvement. The framework's support for reproducible state management and automated evaluation is a practical contribution that could be adopted by the community. The fine-grained breakdown across categories strengthens the diagnostic value beyond aggregate success rates.

major comments (2)

[§3 (Task Dataset)] §3 (Task Dataset): The construction of the 200 manually crafted tasks is presented at a high level without reported validation steps such as human solvability checks, inter-annotator agreement, or pilot runs to confirm that each task has a well-defined, achievable ground-truth outcome. Because the central claims rest on measured failure rates (including the >45% rate for stateful elements), the absence of such validation leaves open the possibility that task formulation itself contributes to the observed difficulties.
[§4.1 (Agentic System and Chrome Extension)] §4.1 (Agentic System and Chrome Extension): The custom extension is described as ensuring consistent initial states across runs, yet no quantitative evaluation of its reliability (e.g., reset success rate, comparison against manual browser resets, or measurement of residual state leakage) is provided. This is load-bearing for the reproducibility of the reported performance numbers and for attributing failures specifically to agent limitations rather than evaluation artifacts.

minor comments (2)

[Abstract] Abstract: The claim of a 'fine-grained analysis' would be clearer if the abstract briefly named the success metric (e.g., task completion rate) and the exact method used to attribute failures to stateful UI elements.
[Related Work] Related Work: The discussion of prior benchmarks (WebArena, SafeArena) could more explicitly contrast the new tasks' focus on legitimate user security/privacy actions versus the safety-against-malicious-action emphasis of existing suites.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and for recognizing the potential impact of WebSP-Eval. We address each major comment point by point below, with clear indications of planned revisions to strengthen the manuscript's rigor and reproducibility.

read point-by-point responses

Referee: §3 (Task Dataset): The construction of the 200 manually crafted tasks is presented at a high level without reported validation steps such as human solvability checks, inter-annotator agreement, or pilot runs to confirm that each task has a well-defined, achievable ground-truth outcome. Because the central claims rest on measured failure rates (including the >45% rate for stateful elements), the absence of such validation leaves open the possibility that task formulation itself contributes to the observed difficulties.

Authors: We agree that additional details on task construction would improve transparency and help readers assess whether formulation contributes to observed failures. Each task was manually designed by the authors with explicit ground-truth action sequences derived from official website documentation and direct UI inspection to ensure a unique, verifiable outcome. Internal pilot testing was performed on a subset of tasks to confirm achievability before full-scale evaluation. We did not conduct formal multi-annotator agreement studies because curation was performed by a small expert team with iterative consensus. In the revised manuscript, we will expand §3 with a dedicated subsection describing the task creation methodology, including concrete examples of task definitions, ground-truth determination, and summary statistics from our internal pilots. This will allow readers to better evaluate the dataset's quality without altering the core results. revision: partial
Referee: §4.1 (Agentic System and Chrome Extension): The custom extension is described as ensuring consistent initial states across runs, yet no quantitative evaluation of its reliability (e.g., reset success rate, comparison against manual browser resets, or measurement of residual state leakage) is provided. This is load-bearing for the reproducibility of the reported performance numbers and for attributing failures specifically to agent limitations rather than evaluation artifacts.

Authors: We acknowledge that quantitative reliability metrics for the extension were not reported, which limits the ability to fully attribute failures to agent capabilities. The extension was implemented to handle deterministic state resets (clearing cookies, local storage, and session data) and account management, and our experimental runs showed consistent behavior with no observed state leakage affecting results. However, we did not include formal measurements such as reset success rates or comparisons to manual resets. In the revised version, we will add a new subsection (or appendix) to §4.1 that provides a more detailed technical description of the extension's architecture and reports any internal reliability checks performed during development. We are also prepared to conduct a small-scale quantitative validation experiment (e.g., measuring reset success over repeated trials) if the referee considers it essential for acceptance. revision: partial

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper introduces an empirical evaluation framework (WebSP-Eval) consisting of a manually crafted dataset of 200 task instances across 28 websites, a custom Chrome extension for state management, and an automated evaluator. All reported results, including the >45% failure rate on stateful UI elements such as toggles and checkboxes, are direct measurements obtained by executing 8 agent instantiations on these tasks. No mathematical derivations, fitted parameters renamed as predictions, uniqueness theorems, or ansatzes appear in the provided text. The central claims rest on independent empirical observations rather than any reduction to the paper's own inputs or self-citations.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper relies on standard assumptions in benchmark creation for AI agents. No free parameters or invented entities are introduced; the work extends existing multimodal LLM agent architectures with a custom state-management extension.

axioms (1)

domain assumption The manually crafted 200 tasks across 28 websites represent typical real-world user security and privacy interactions.
This underpins the claim that the benchmark measures relevant agent capabilities.

pith-pipeline@v0.9.0 · 5559 in / 1431 out tokens · 64660 ms · 2026-05-10T18:44:14.179559+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability
cs.CL 2026-05 unverdicted novelty 4.0

The paper develops a unified framework that organizes computer-use agent reliability around perception-decision-execution layers and creation-deployment-operation-maintenance stages to map security and alignment inter...