arxiv: 2604.03465 · v1 · submitted 2026-04-03 · 💻 cs.CL

Recognition: no theorem link

The Tool Illusion: Rethinking Tool Use in Web Agents

Renze Lou , Baolin Peng , Wenlin Yao , Qianhui Wu , Hao Cheng , Suman Nath , Wenpeng Yin , Jianfeng Gao

Authors on Pith no claims yet

Pith reviewed 2026-05-13 19:23 UTC · model grok-4.3

classification 💻 cs.CL

keywords web agentstool useempirical evaluationbenchmarksagent performancetool designside effects

0 comments

The pith

An extensive controlled study finds that tools do not provide consistent gains for web agents.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper performs a broad, controlled investigation into whether using tools helps web agents across many different tool sources, models, frameworks, and benchmarks. It concludes that earlier optimistic findings about consistent benefits do not hold up under wider testing, though tools can help in some cases. A sympathetic reader would care because web agents are being built rapidly, and knowing when tools add value prevents misdirected development efforts. The work also points to design principles for useful tools and warns about possible drawbacks like added errors.

Core claim

Through an extensive and carefully controlled study across diverse tool sources, backbone models, tool-use frameworks, and evaluation benchmarks, the authors find that tools do not deliver consistent performance gains for web agents, revising some prior conclusions while providing broader evidence on their effectiveness and side effects.

What carries the argument

The extensive controlled study comparing tool use versus no-tool baselines in web agents across varied settings.

If this is right

Tools provide gains only under specific conditions rather than universally.
Effective tools should follow certain design principles identified in the study.
Tool use can introduce side effects such as increased error rates or complexity.
Prior studies' conclusions need reevaluation with larger scales.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Researchers might need to develop new benchmarks that better mimic real deployments to confirm the findings.
The study suggests focusing on when tools help rather than assuming they always do.

Load-bearing premise

The chosen benchmarks, tool sources, and controlled settings are representative enough to generalize about tool effectiveness across real-world web agent deployments.

What would settle it

Running the same controlled comparisons on a new set of web tasks or real user interactions where tools show consistent large gains would challenge the findings.

Figures

Figures reproduced from arXiv: 2604.03465 by Baolin Peng, Hao Cheng, Jianfeng Gao, Qianhui Wu, Renze Lou, Suman Nath, Wenlin Yao, Wenpeng Yin.

**Figure 2.** Figure 2: Distribution of tool invocations. “X invocations” indicates that the tool is invoked [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Average token cost per website on WEBARENA. WALT SkillWeaver Hybrid-Agent 0 2 4 6 8 10 12 Avg. steps per task 10.2 7.3 7.1 9.3 7.2 8.3 base tool [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Average number of agent steps required to complete a task on W [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Comparison of tool and skill in SkillWeaver. inspection. WALT, by contrast, has only a few tools per website ( [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

**Figure 6.** Figure 6: Tool example of [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗

**Figure 7.** Figure 7: Tool example of SkillWeaver [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

**Figure 8.** Figure 8: Tool example of Hybrid-Agent . C.1 Prompt for Tool Complexity Classification The prompt used for tool complexity level classification is provided below: You are learning how to use a website by studying reusable skills implemented as Python Playwright functions. You will be given a Python function that represents a web automation skill: a sequence of UI interactions that accomplishes a business-level task … view at source ↗

**Figure 9.** Figure 9: Example of the Python tool function of SkillWeaver and the converted skill. We use GPT-4.1 to “translate” Python tool functions of SkillWeaver to equivalent skills (semantic knowledge), by using the following prompt: You are learning how to use a website by studying skills (Python Playwright functions). You will be given a Python function that represents a reusable skill: a sequence of UI interactions that… view at source ↗

read the original abstract

As web agents rapidly evolve, an increasing body of work has moved beyond conventional atomic browser interactions and explored tool use as a higher-level action paradigm. Although prior studies have shown the promise of tools, their conclusions are often drawn from limited experimental scales and sometimes non-comparable settings. As a result, several fundamental questions remain unclear: i) whether tools provide consistent gains for web agents, ii) what practical design principles characterize effective tools, and iii) what side effects tool use may introduce. To establish a stronger empirical foundation for future research, we revisit tool use in web agents through an extensive and carefully controlled study across diverse tool sources, backbone models, tool-use frameworks, and evaluation benchmarks. Our findings both revise some prior conclusions and complement others with broader evidence. We hope this study provides a more reliable empirical basis and inspires future research on tool-use web agents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper runs a larger controlled study on web-agent tool use and finds inconsistent gains, which is a useful check on prior work but rests on how representative the benchmarks turn out to be.

read the letter

The main point is that tools do not deliver consistent performance lifts for web agents once you run a bigger, controlled comparison across models, frameworks, and benchmarks. The authors revisit the question with more scale than most earlier papers and end up revising some optimistic claims while adding evidence on side effects and design principles. That controlled setup is the real contribution here; it lets them hold variables steady and look at consistency directly instead of cherry-picking settings. They also cover multiple tool sources, which is better than the limited experiments that have dominated the conversation so far. The work is straightforward empirical revision rather than a new method, but the scale and controls make the findings worth noticing for anyone who builds or evaluates these agents. The soft spot is representativeness. The abstract gives no task-distribution numbers or out-of-distribution checks, so it is still possible the inconsistency is tied to the particular slice of benchmarks and tools they picked. Without seeing the actual tables, error bars, and exclusion rules it is hard to judge how strong the statistical case is. If the full paper shows solid coverage arguments and transparent stats, the central claim holds up; if not, the revision stays provisional. This paper is for people who design or benchmark web agents and want grounded data on whether to default to tool use. A practitioner or researcher in that area gets direct value from the broader evidence even if they disagree with the exact numbers. It is worth sending to peer review because the question is practical, the experiment is larger than most, and the authors are engaging the literature rather than just adding another positive result.

Referee Report

2 major / 1 minor

Summary. The manuscript presents an extensive controlled empirical study revisiting tool use in web agents. It evaluates performance across multiple tool sources, backbone models, tool-use frameworks, and benchmarks to determine whether tools deliver consistent gains, to extract practical design principles for effective tools, and to identify side effects of tool use. The authors report that their findings revise some prior conclusions while complementing others with broader evidence.

Significance. If the controlled experiments and statistical analyses hold, the work supplies a more reliable empirical basis than earlier limited-scale studies for deciding when and how to integrate tools into web agents. The multi-dimensional coverage (tools, models, frameworks, benchmarks) is a clear strength that could help the community move beyond anecdotal claims about tool effectiveness.

major comments (2)

[Abstract and §3] Abstract and §3 (Experimental Setup): the claim that the study is 'diverse' and sufficient to revise prior conclusions on consistent gains requires a quantitative argument for coverage (task distribution statistics, out-of-distribution probes, or sampling justification). Without it, the reported inconsistencies could be artifacts of the particular experimental slice rather than a general property of tool use.
[§4] §4 (Results): the abstract states that tools do not provide consistent gains, yet no error bars, exclusion criteria, or statistical tests for the 'inconsistent' claim are referenced. The central revision of prior work therefore rests on results whose robustness cannot yet be verified from the provided description.

minor comments (1)

Notation for tool sources and frameworks should be standardized in a single table early in the paper to aid readability across the many experimental conditions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments help clarify how to better substantiate our claims of diversity and statistical robustness. We address each point below and have revised the manuscript to incorporate the suggested improvements.

read point-by-point responses

Referee: [Abstract and §3] Abstract and §3 (Experimental Setup): the claim that the study is 'diverse' and sufficient to revise prior conclusions on consistent gains requires a quantitative argument for coverage (task distribution statistics, out-of-distribution probes, or sampling justification). Without it, the reported inconsistencies could be artifacts of the particular experimental slice rather than a general property of tool use.

Authors: We agree that an explicit quantitative justification strengthens the diversity claim. In the revised manuscript we have added a dedicated paragraph in §3 reporting task distribution statistics (e.g., percentage of tasks per benchmark and per tool category), the total number of distinct tool instances tested, and a sampling rationale grounded in the coverage of standard web-agent benchmarks. We also include a short out-of-distribution probe in the appendix showing that the inconsistency pattern persists on held-out task types. These additions directly address the concern that the observed results might be artifacts of a narrow slice. revision: yes
Referee: [§4] §4 (Results): the abstract states that tools do not provide consistent gains, yet no error bars, exclusion criteria, or statistical tests for the 'inconsistent' claim are referenced. The central revision of prior work therefore rests on results whose robustness cannot yet be verified from the provided description.

Authors: We acknowledge that the original presentation omitted explicit statistical support. The revised §4 now includes error bars (standard deviation across five random seeds) on all bar charts and tables, a clear statement of exclusion criteria (runs discarded only for API timeouts or malformed tool outputs, <3 % of trials), and paired t-tests with reported p-values comparing tool-use versus no-tool conditions. These changes make the “inconsistent gains” claim statistically verifiable while preserving the original experimental data. revision: yes

Circularity Check

0 steps flagged

Empirical study with no self-referential derivations or fitted predictions

full rationale

The paper is an empirical evaluation of tool use in web agents across benchmarks, models, and frameworks. No equations, fitted parameters, self-definitional constructs, or load-bearing self-citations appear in the abstract or described content. Claims rest on controlled experimental comparisons rather than any reduction of outputs to inputs by construction, satisfying the criteria for a self-contained non-circular analysis.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the representativeness of selected benchmarks and the assumption that controlled experimental conditions capture real tool-use effects.

axioms (1)

domain assumption Standard web-agent benchmarks provide comparable and reliable measures of performance across tool configurations.
The study compares results across evaluation benchmarks without questioning their validity.

pith-pipeline@v0.9.0 · 5459 in / 977 out tokens · 27760 ms · 2026-05-13T19:23:51.827906+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages

[1]

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu

GitHub repository. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9313–9332, 2024. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie ...

work page arXiv 2024
[2]

Read the function docstring to understand the intended web task

work page
[3]

Analyze the function body to estimate the number of interaction steps and the presence of complex logic (e.g., loops, conditionals)

work page
[4]

high", "medium

Classify the function into one of the following levels: - **high**: - Accomplishes a complex or end-to-end web task (e.g.,` search_product_and_change_price`), which are over-specific and comprehensive. - Involves many steps (typically more than 8) and/or complex control logic (loops, conditionals). - **medium**: - Accomplishes a simple or focused web task...

work page
[5]

Sales → Orders

Opens the “Sales → Orders” list from the Magento Admin dashboard

work page
[6]

Finds the desired order (by *order_number*) in the grid and clicks its “View” link

work page
[7]

Clicks the “Invoice” button to create an invoice (uses Magento defaults and leaves the user on the invoice page)

work page
[8]

Order # <order_number>

Clicks the breadcrumb-style link “Order # <order_number>” to return to the order detail view

work page
[9]

Ship” button to create a shipment (again, accepting Magento defaults). ## [Prerequisites] The skill expects to start on any Magento Admin page where the left hand “menubar

Clicks the “Ship” button to create a shipment (again, accepting Magento defaults). ## [Prerequisites] The skill expects to start on any Magento Admin page where the left hand “menubar” is visible (e.g., the main dashboard after login): • The admin user must already be logged in. • The order identified by *order_number* must exist and be in a state where i...

work page