AutoSurfer -- Teaching Web Agents through Comprehensive Surfing, Learning, and Modeling
Pith reviewed 2026-05-07 10:17 UTC · model grok-4.3
The pith
AutoSurfer generates more complete web trajectories by using breadth-first exploration and path-guided task synthesis, leading to higher agent accuracy on benchmarks.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
AutoSurfer employs a systematic breadth-first exploration strategy that maintains a queue of discovered pages and action traces, propagates knowledge across pages to avoid redundant exploration, and recursively expands multi-level graphical user interface elements. It then leverages the exploration trajectory to guide task synthesis, reducing hallucinations by grounding complex tasks in actual navigation paths rather than isolated actions or page content alone. The same trajectories serve as hints to steer a web agent toward more accurate and reliable trajectory refinement. Together these steps enable comprehensive coverage of a website's action space and produce data suitable for training,
What carries the argument
AutoSurfer's three-part pipeline of breadth-first exploration with knowledge propagation, trajectory-guided task synthesis, and hint-based refinement.
If this is right
- Websites can be explored with less redundancy through queued page tracking and knowledge sharing.
- Task synthesis becomes grounded in actual navigation paths, lowering hallucination rates.
- Trajectory refinement improves when guided by the same exploration data used for synthesis.
- Fine-tuned agents achieve higher overall task completion rates on standard benchmarks.
- The resulting task set shows a more diverse distribution than outputs from prior generation methods.
Where Pith is reading between the lines
- This style of exploration may lower the amount of human effort needed to prepare training data for new websites.
- The same pipeline could extend to generating data for agents in other interactive settings such as mobile apps or desktop software.
- Systematic coverage of action spaces might improve generalization when agents move between related sites.
- Grounding synthesis in real paths could become a standard step in creating reliable synthetic data for agent training.
Load-bearing premise
That breadth-first exploration plus trajectory-guided synthesis and refinement produces trajectories that are both more complete and less hallucinated than prior methods and that these transfer to improved fine-tuning performance on held-out tasks.
What would settle it
If fine-tuning the same multimodal model on AutoSurfer-generated trajectories yields no accuracy improvement or a drop versus data from Explorer, OS-Genesis, or SynthAgent when tested on the same WebArena tasks, the central claim would be falsified.
Figures
read the original abstract
Recent advances in multimodal large language models (LLMs) have revolutionized web agents that can automate complex tasks on websites. However, their accuracy remains limited by the scarcity of high-quality web trajectory training data. Existing automatic trajectory generation methods suffer from incomplete website coverage due to homepage-based task proposals or random-walk exploration. Such methods often result in hallucinated or ambiguous task synthesis that lead to incomplete and unreliable trajectory generation. Here, we present AutoSurfer, a comprehensive web trajectory generator that addresses these limitations through three key innovations. First, AutoSurfer employs a systematic breadth-first exploration strategy that maintains a queue of discovered pages and action traces, propagates knowledge across pages to avoid redundant exploration, and recursively expands multi-level graphical user interface elements - closely resembling how a human would learn a new website. Second, AutoSurfer leverages the exploration trajectory to guide task synthesis, reducing hallucinations by grounding complex tasks in actual navigation paths rather than isolated actions or page content alone. Third, AutoSurfer uses the same exploration trajectory as hints to steer a web agent toward more accurate and reliable trajectory refinement. Together, these innovations enable AutoSurfer to comprehensively cover a website's action space and generate data suitable for training website-specific LLMs. We evaluate AutoSurfer on the WebArena benchmark by fine-tuning Qwen2.5-VL-7B-Instruct and demonstrate that it outperforms state-of-the-art methods - Explorer, OS-Genesis, and SynthAgent - achieving up to 24.23% overall task completion accuracy compared to 19.59% for the best prior method. Further, task diversity analysis demonstrates that AutoSurfer yields a more diverse distribution of synthesized tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces AutoSurfer, a web trajectory generator that uses breadth-first exploration (with queue-based page discovery and recursive GUI expansion), trajectory-guided task synthesis to reduce hallucinations, and trajectory-guided refinement to produce more reliable trajectories. These are used to fine-tune Qwen2.5-VL-7B-Instruct, which is then evaluated on the WebArena benchmark and reported to reach 24.23% overall task completion accuracy versus 19.59% for the strongest prior method (Explorer/OS-Genesis/SynthAgent), with additional claims of greater task diversity.
Significance. If the performance delta is shown to be robust and causally attributable to the three innovations, the work would provide a concrete advance in automatic generation of high-quality, grounded web-agent training data, addressing a recognized bottleneck in scaling multimodal web agents beyond limited human-curated trajectories.
major comments (2)
- [Abstract / method description] Abstract and method description: the central claim that breadth-first exploration plus trajectory-guided synthesis/refinement yields verifiably more complete and less hallucinated trajectories (which then transfer to the 4.64-point accuracy gain) is asserted without any supporting quantitative measurements such as action-coverage statistics, hallucination-rate comparisons, or human-rated groundedness scores on the generated trajectories versus baselines.
- [Abstract] Abstract: performance numbers (24.23% vs. 19.59%) are stated without an experimental protocol, description of task hold-out procedure, statistical significance tests, error bars, or details on how the three baselines were re-implemented or fine-tuned under identical conditions, rendering it impossible to assess whether the gain is robust or sensitive to post-hoc choices.
minor comments (2)
- [Evaluation] The diversity analysis is mentioned but not quantified (e.g., no entropy, coverage, or statistical comparison metrics); adding a table or figure with explicit diversity measures would strengthen the supporting claim.
- [Method] Notation for the exploration queue, action traces, and refinement hints is introduced without a clear pseudocode or diagram; a single figure illustrating the three-stage pipeline would improve readability.
Simulated Author's Rebuttal
We thank the referee for their thoughtful and constructive comments. We address each major comment point by point below, indicating where revisions have been made to the manuscript.
read point-by-point responses
-
Referee: [Abstract / method description] Abstract and method description: the central claim that breadth-first exploration plus trajectory-guided synthesis/refinement yields verifiably more complete and less hallucinated trajectories (which then transfer to the 4.64-point accuracy gain) is asserted without any supporting quantitative measurements such as action-coverage statistics, hallucination-rate comparisons, or human-rated groundedness scores on the generated trajectories versus baselines.
Authors: We agree that direct quantitative metrics would provide stronger support for the trajectory quality claims. In the revised manuscript we have added action-coverage statistics (new Table 2) and hallucination-rate comparisons (Section 4.3) showing AutoSurfer achieves 18% higher coverage and 12% lower hallucination rates than the strongest baseline. Human-rated groundedness scores were not collected owing to the prohibitive annotation cost; we instead rely on the automated metrics together with the downstream 4.64-point WebArena gain as evidence of improved trajectory reliability. revision: partial
-
Referee: [Abstract] Abstract: performance numbers (24.23% vs. 19.59%) are stated without an experimental protocol, description of task hold-out procedure, statistical significance tests, error bars, or details on how the three baselines were re-implemented or fine-tuned under identical conditions, rendering it impossible to assess whether the gain is robust or sensitive to post-hoc choices.
Authors: The abstract is space-constrained, but the full experimental protocol—including the standard WebArena task hold-out, identical fine-tuning of all baselines on Qwen2.5-VL-7B-Instruct, and statistical significance testing—is provided in Sections 5.1–5.2. We have revised the abstract to note that results are averaged over three runs with standard deviations and refer readers to the main text for the complete protocol and p-values. revision: yes
Circularity Check
No circularity: empirical claims rest on external benchmark evaluation
full rationale
The paper describes a procedural pipeline (breadth-first exploration maintaining page/action queues, trajectory-guided task synthesis, and trajectory-hinted refinement) without any equations, fitted parameters, or self-referential definitions. The claimed superiority is demonstrated solely via fine-tuning Qwen2.5-VL-7B-Instruct on the generated trajectories and measuring task-completion accuracy on the held-out WebArena benchmark, with numeric comparisons to three independently published prior methods. No self-citations are used to justify uniqueness theorems, ansatzes, or load-bearing premises; no predictions reduce to inputs by construction; and no renaming of known results occurs. The derivation chain is therefore self-contained as an algorithmic contribution whose outputs are externally falsifiable rather than tautological.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
•Include elements that are viewed as pictures or icons
Examine every annotated element: • For each HTML element with an “id” attribute, determine if it can be used to perform a short task. •Include elements that are viewed as pictures or icons
-
[3]
Search for an item
Respect action order in short tasks: • Some short tasks will require multiple actions to be completed in a specific order. As such, these actions should be collapsed into a single short task with an ordered action sequence. •Example 1: “Search for an item” short task is completed by (1) filling the search box and (2) pressing ENTER. • Example 2: “Create p...
-
[4]
• When a short task contains one or more fill actions, ensure that the short task ends with a non-fill action (e.g., click, press)
Appropriate usage of multiple actions in short tasks: • A short task can have multiple actions when they appear as a group in the UI to perform a specific function. • When a short task contains one or more fill actions, ensure that the short task ends with a non-fill action (e.g., click, press). • A help or tooltip or formatting hint element should not be...
-
[5]
•For a fill action, provide realistic text input that a user would typically enter in that field
Use realistic inputs: •Predict meaningful input values based on the context of the screenshot and HTML. •For a fill action, provide realistic text input that a user would typically enter in that field. • Carefully examine whether an input field can contain space characters, special characters, or only numbers, and provide input accordingly
-
[6]
•Use false if the short task is already present in the list of observed short tasks
Setis allowedappropriately: • Use false for short tasks that require login, logout, sign up, account creation/modification, payment, or operate on transient UI elements like ads, pop-ups, or modals. •Use false if the short task is already present in the list of observed short tasks. •However, account viewing is allowed. Use true for viewing account or profile
-
[7]
task list
Avoid some tasks and actions: •Avoid any read text actions. •Avoid any short task that operates on date pickers. You are provided the following information: • The screenshot of the web page with visible HTML elements annotated with red/yellow circles and numbers. The screenshot is attached with this message. •The simplified HTML of the screenshot:{{HTML}}...
-
[8]
•Include elements that are viewed as pictures or icons
Examine every annotated element from the second screenshot: • For each HTML element with an “id” attribute, determine if it can be used to perform a short task. •Include elements that are viewed as pictures or icons
-
[9]
• First, identify short tasks that cover panel/header/sidebar elements, typically located at the top or left or right side of the screenshot
Generate diverse short tasks: • Identify as many short tasks as possible as long as they can perform meaningful actions on the web page. • First, identify short tasks that cover panel/header/sidebar elements, typically located at the top or left or right side of the screenshot. • Second, identify short tasks for expandable UI elements, such as menus, drop...
-
[10]
Search for an item
Respect action order in short tasks: • Some short tasks will require multiple actions to be completed in a specific order. As such, these actions should be collapsed into a single short task with an ordered action sequence. • For example, “Search for an item” short task is completed by (1) filling the search box and (2) pressing ENTER. • Some other short ...
-
[11]
Use realistic inputs: •Predict meaningful input values based on the context of the screenshot and HTML
-
[12]
•However, account viewing is allowed
Setis allowedappropriately: • Use false for short tasks that require login, logout, sign up, account creation/modification, payment, or operate on transient UI elements like ads, pop-ups, or modals. •However, account viewing is allowed. Use true for viewing account or profile
-
[13]
task list
Avoid some tasks and actions: •Avoid any read text actions. •Avoid any short task that operates on date pickers. You are provided the following information: • Two screenshots of the web page where first one is before the action and the second one is after the action. The screenshots are attached with this message. •The simplified HTML of the second screen...
-
[14]
fixed” if it has fixed position on the page and provides specific functionality. For example, “search box
An element is “fixed” if it has fixed position on the page and provides specific functionality. For example, “search box” at Amazon home page is “fixed”
-
[15]
dynamic” if it does not have a fixed position on the page. Most commonly, “dynamic
An element is “dynamic” if it does not have a fixed position on the page. Most commonly, “dynamic” elements are list of items, such as product lists and search results, providing similar functionalities. For example, “product list” at Amazon search results page is “dynamic”
-
[16]
If a list of UI elements appear as a group then each UI element in that group should be assigned the same group number
Assign a group number for each dynamic UI element. If a list of UI elements appear as a group then each UI element in that group should be assigned the same group number. For example, if there are 20 products from a search result in a shopping website, then all those 20 product UI elements should be assigned the same group number
-
[17]
As such, each group should have a different group number
There can be multiple groups of dynamic UI elements on the same page. As such, each group should have a different group number. For example, if there are 20 products from a search result and 10 recommended products on the same page, then the 20 products should be assigned one group number and the 10 recommended products should be assigned another group number
-
[18]
Sometimes, there are UI elements that appear closer in a group but perform quite different actions/functions
Short tasks in each dynamic group should perform similar actions/functions. Sometimes, there are UI elements that appear closer in a group but perform quite different actions/functions. In such cases, those UI elements should be assigned different group numbers. For example, in a list of product search results, there are links to product details and butto...
-
[19]
task list
Usually following UI elements are not dynamic: •UI elements located at top or left or right header/panel/sidebar. •Expandable UI elements, such as menus, dropdowns, and down-arrow buttons. •UI elements appeared as a list of tabs. You are provided the following information: • The screenshot of the web page with visible HTML elements annotated with red/yell...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.