Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WebChoreArena adds 532 tedious web tasks to WebArena and finds that even the strongest tested agent, Gemini 2.5 Pro, solves fewer than half of them.

desk verdict A genuinely useful, reproducible benchmark extension of WebArena; the missing human baseline and Claude-based task refinement are real but fixable gaps. read the letter →

arxiv 2506.01952 v1 pith:MOOTNZUQ submitted 2025-06-02 cs.CL cs.AIcs.LG

Atsuyuki Miyai , Zaiying Zhao , Kazuki Egashira , Atsuki Sato , Tatsumi Sunada , Shota Onohara , Hiromasa Yamanishi , Mashiro Toyooka
show 4 more authors
Kunato Nishina Ryoma Maeda Kiyoharu Aizawa Toshihiko Yamasaki
This is my paper
classification cs.CLcs.AIcs.LG
keywords webbrowsingagentsLLMevaluationbenchmarkArenamemory-intensivetasksreproducibilityautomationlong-horizon
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WebChoreArena is a new benchmark of 532 human-curated tasks inside the four WebArena websites, aimed at chores that require holding large amounts of page content in memory, calculating over that content, or retaining information across multiple pages. The paper's central claim is that these tasks separate LLM-based agents far more sharply than the original WebArena tasks: GPT-4o drops from 42.8% to no more than 6.8%, while the strongest tested configuration, Gemini 2.5 Pro with BrowserGym, reaches 44.9%. If the claim is right, the field gains a harder, fully reproducible yardstick for tracking agent progress and for identifying which mechanism, whether memory, arithmetic, or visual reading, limits a given agent. The authors also report that about 134 of the 684 non-map WebArena tasks they inspected contain annotation or evaluation errors, which they offer as the motivation for building a cleaner, more demanding suite.

What carries the argument

The carrying mechanism is the task-design pipeline and its output: 117 templates expanded into 532 concrete tasks, each constructed so that the correct answer requires memory at scale, arithmetic over remembered numbers, cross-page recall, or a website-specific operation. Evaluation reuses WebArena's three checks, exact and partial string matching, final URL matching, and programmatic HTML state verification, so success requires either the exact expected output or the exact expected page state. The shared four-site environment makes scores directly comparable to WebArena and lets the benchmark isolate the added difficulty of the new tasks.

What would settle it

Have independent human operators attempt all 532 tasks under the same browser conditions and without model assistance; if a meaningful fraction of tasks yields human disagreement with the provided ground truth, or if human success is far below near-ceiling, the reported model scores mix task ambiguity with agent capability and the benchmark's main claim loses force.

Watch

Extended reading notes

Core claim

On its own terms, the paper discovers that the current bottleneck for web agents is not general navigation but the sustained memory and arithmetic demanded by tedious chores. Across two agent harnesses and three LLMs, every model loses roughly 14 to 36 percentage points relative to its WebArena score; the best result is 44.9% (BrowserGym with Gemini 2.5 Pro), and the worst is 2.6% (BrowserGym with GPT-4o). The 532 tasks are spread over Shopping, Shopping Admin, Reddit, GitLab, and cross-site scenarios, and each task carries an input-modality requirement plus up to two of four types: Massive Memory, Calculation, Long-Term Memory, and Others. Because tasks were triple-annotated per site and iteratively revised against agent failures, the authors interpret the drop as measuring task difficulty rather than annotation noise.

Load-bearing premise

The load-bearing premise is that every one of the 532 tasks has a unique, reachable correct answer reflected in the ground truth, and that refining tasks with a Claude-based agent did not bias the suite for or against any model family.

Editorial extensions

If this is right

  • GPT-4o scores 2.6–6.8% on WebChoreArena versus 36.4–42.8% on WebArena, so a high WebArena score no longer implies competence at memory-heavy chores.
  • The model spread widens from roughly 23 points on WebArena (with BrowserGym, GPT-4o at 36.4% and Gemini 2.5 Pro at 59.2%) to roughly 42 points on WebChoreArena (2.6% versus 44.9%), which the paper presents as clearer signal for ranking models.
  • Even the best configuration leaves a double-digit gap relative to its WebArena performance, so the benchmark does not appear saturated.
  • Type-level results link failures to agent mechanisms, such as note-based memory underperforming on Massive Memory tasks, giving developers a targeted place to improve.
  • Adding screenshots to the accessibility-tree input generally lowered accuracy, especially on tasks that can only be solved from text, suggesting multimodal integration is not yet a strength.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that a human baseline would test whether this difficulty is intrinsic to the chores or specific to current models; collecting one is a natural extension.
  • If the benchmark is adopted broadly, agent rankings may increasingly reward memory-management design and arithmetic reliability rather than navigation skill alone.
  • The near-zero effect of giving agents a calculator suggests the bottleneck is not tool availability but the agent's decision to use it, pointing to prompt or training interventions that make tool use routine.
  • Because the suite shares WebArena's environments and formats, existing agent code can be replayed with minimal changes, so the same 532 tasks could serve as a longitudinal tracker of model progress.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces WebChoreArena, a benchmark of 532 human-curated tasks built on the four WebArena simulation environments (Shopping, Shopping Admin, Reddit, GitLab, plus Cross-site tasks). The tasks are designed to be tedious and memory-intensive, and are categorized into Massive Memory, Calculation, Long-Term Memory, and Others. The authors evaluate GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro with two open-source agents, AgentOccam and BrowserGym, reporting large performance drops relative to WebArena and a wider spread between model families. They conclude that WebChoreArena more clearly distinguishes LLM-based web agents and still leaves substantial room for improvement.

Significance. If the benchmark is valid, it is a potentially valuable resource: it extends the widely used WebArena environment, preserves reproducibility, introduces a task-type taxonomy that targets memory and calculation skills, and provides detailed error analyses plus open agents. The authors also contribute a useful review of annotation errors in WebArena (Appendix A.1). However, the current submission lacks the external validation needed to support the central claim that the low observed accuracies reflect agent capability rather than unsolvable or ambiguous tasks.

major comments (3)
  1. [Sec. 3.4 / Table 1] The benchmark's validity rests on the assumption that all 532 tasks have solvable, unambiguous ground truth, but the manuscript provides no human baseline or independent audit. The only validation is internal cross-checking by author-annotators (Sec. 3.4), and Appendix A.1 itself reports that 134/684 (about 20%) of WebArena tasks contain annotation or evaluation errors. Since WebChoreArena tasks are deliberately longer and more memory-intensive, the prior probability of hidden ambiguity is higher, not lower. Without a human ceiling, the headline claim that Gemini 2.5 Pro's 44.9% (BrowserGym) indicates 'significant room for improvement' (F2, Sec. 5.1) cannot be distinguished from the alternative that the effective solvability ceiling of the benchmark is near that score. Please add a human performance baseline on a stratified sample (e.g., 50-100 tasks) or an independent audit with inter-annotator agreement, and report task-level solvability.
  2. [Sec. 3.4] The construction pipeline introduces a possible selection bias: task ideas were prototyped and refined using a Claude-based agent to 'identify model limitations and refine the task designs,' and Claude 3.7 Sonnet is then one of the evaluated models. This creates a feedback loop in which the benchmark's difficulty is partly defined by Claude's failure modes, which can inflate the measured performance gap between Claude and other models (F3, Sec. 5.1) even if no single task is circular. The paper does not report how many tasks were revised or discarded because of Claude feedback, nor whether the same tasks would have been selected without model-in-the-loop refinement. Please quantify the revision process and either include a held-out set of tasks constructed without agent-in-the-loop feedback or show through a sensitivity analysis that the model ranking is stable across task subsets.
  3. [Sec. 5.1, F3 / Table 1] The central claim that WebChoreArena 'more clearly demonstrates performance differences between LLMs than WebArena' is based on point estimates from a single run (Table 1) and on 532 task instances that are not independent: Sec. 3.4 states the tasks are generated from only 117 templates. No confidence intervals, repeated runs, or significance tests are reported, so the larger observed spread (GPT-4o 2.6% vs. Gemini 2.5 Pro 44.9% with BrowserGym, versus 36.4% vs. 59.2% on WebArena) could be partly a floor/ceiling artifact or template-level effects rather than a stable property of the benchmark. Please report bootstrap confidence intervals clustered by template, or otherwise test whether the model gap is statistically significant and robust to template removal.
minor comments (6)
  1. [Sec. 3.1 / Sec. 3.2] Section 3.1 mentions the map website as part of WebArena, while Section 3.2 lists only four websites; the exclusion of the map domain should be stated earlier and more prominently, since the reasons given in Sec. 3.4 and Appendix A.2 are substantive.
  2. [Sec. 5.1, F2] The phrase 'approximately 14% performance drop' is ambiguous; specify that it is an absolute percentage-point drop and state which agent configuration is being compared (AgentOccam, 17.0 points, or BrowserGym, 14.3 points).
  3. [Table 3] The table header 'Input Text(#15)' is confusing because the column appears to label the required observation type rather than the model input; rename the columns to 'Required Obs: Text' and 'Required Obs: Any' to match the text.
  4. [Sec. 5.2, Table 4] The statement that 'the overall performance remained largely unchanged' glosses over notable differences: GPT-4o drops from 3.7% to 2.8%, while Gemini 2.5 Pro rises from 40.0% to 42.8%; given the small number of tool-using tasks (35, 59, and 41), report whether these differences are within expected variability.
  5. [Sec. 6] The error analysis is anecdotal; it would be more informative to quantify the frequency of each error type across Gemini 2.5 Pro's failures rather than listing representative examples.
  6. [Sec. 3.5] The fuzzy_match evaluator uses GPT-4o, but GPT-4o is also an evaluated model; clarify whether the judge version matches the evaluated version and whether any validation of the judge's agreement with human judgments was performed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; benchmark scores are measured outcomes rather than constructed equivalences.

full rationale

WebChoreArena is a dataset-and-evaluation benchmark: there are no fitted parameters, no equations that reduce one quantity to another, and no self-citation chain that supplies the load-bearing content. The central claims—that GPT-4o drops to 2.6–6.8%, that Gemini 2.5 Pro remains near 44.9%, and that WebChoreArena separates models more sharply than WebArena—are supported by the reported execution results in Tables 1–4 rather than by construction. The closest candidate is Sec. 3.4, where a Claude-based agent was used to prototype and refine task designs before Claude 3.7 Sonnet was later evaluated on the tasks. That is a selection-bias or overfitting concern for cross-model difficulty comparisons, but it is not circular: task difficulty was not defined as 'Claude fails this task', and the reported accuracies come from fresh agent runs whose correctness is checked against ground truth. Similarly, using GPT-4o as the fuzzy_match evaluator while GPT-4o is one of the evaluated models is a potential evaluation bias, but it does not make the accuracy numbers equal to the benchmark's inputs by definition. The absence of a human baseline and of an independent audit is a validity and correctness risk, not a circularity. No load-bearing self-citations or imported uniqueness theorems are present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark inherits assumptions from WebArena and from the authors' annotation protocol. No numeric free parameters are fitted; the main assumptions are about environment fidelity, ground-truth quality, and the neutrality of the Claude-based task refinement step.

assumptions (4)
  • domain assumption WebArena's four simulated websites faithfully represent real web browsing behavior and observations.
    WebChoreArena inherits all environments from WebArena (Sec 3.2); if the simulators diverge from real sites, task validity and the realism of the accuracy numbers weaken.
  • domain assumption The 532 tasks have unique, correct ground-truth answers.
    Sec 3.4 relies on three annotators per website and iterative agent runs, but no human baseline or external audit is reported, so solvability and answer uniqueness are assumed.
  • ad hoc to paper Task refinement against a Claude-based agent does not bias cross-model comparisons.
    Sec 3.4 states tasks were prototyped and refined by evaluating a Claude-based agent to identify model limitations; this construction choice is not controlled for when comparing GPT-4o, Claude, and Gemini.
  • domain assumption Accessibility-tree observations are sufficient for tasks labeled 'text' or 'any'.
    Task modality annotations (Sec 3.3, Appendix B) assume the tree contains the information needed and that screenshots are only necessary for the 12 image-only tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks." pith.science (2026). https://pith.science/paper/MOOTNZUQ

@misc{pith2026250601952,
  author       = {Pith},
  title        = {Pith review of: WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOOTNZUQ}},
  note         = {Machine review of arXiv:2506.01952}
}
read the original abstract

Powered by a large language model (LLM), a web browsing agent operates web browsers in a human-like manner and offers a highly transparent path toward automating a wide range of everyday tasks. As web agents become increasingly capable and demonstrate proficiency in general browsing tasks, a critical question emerges: Can they go beyond general browsing to robustly handle tasks that are tedious and complex, or chores that humans often avoid doing themselves? In this paper, we introduce WebChoreArena, a new fully reproducible benchmark comprising 532 carefully curated tasks designed to extend the scope of WebArena beyond general browsing to more labor-intensive and tedious tasks. WebChoreArena systematically integrates three key challenges: (i) Massive Memory tasks requiring accurate retrieval of large amounts of information in the observations, (ii) Calculation tasks demanding precise mathematical reasoning, and (iii) Long-Term Memory tasks necessitating long-term memory across multiple webpages. Built on top of the fully reproducible and widely adopted four WebArena simulation environments, WebChoreArena ensures strict reproducibility and enables fair, direct comparisons with the established WebArena benchmark, offering key insights into agent progress. Our experimental results demonstrate that as LLMs evolve, represented by GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro, significant improvements in performance are observed on WebChoreArena. These findings suggest that WebChoreArena is well-suited to measure the advancement of state-of-the-art LLMs with greater clarity. Nevertheless, the results also indicate that even with Gemini 2.5 Pro, there remains substantial room for improvement compared to WebArena, highlighting the increased challenges posed by WebChoreArena.

Figures

Figures reproduced from arXiv: 2506.01952 by the authors.

Figure 1
Figure 1. The WebChoreArena challenge. WebChoreArena extends WebArena by introducing more complex and labor-intensive tasks, pushing the boundaries of agent capabilities. This enhanced benchmark allows for a clearer evaluation of progress in advanced models and reveals that even powerful models such as Gemini 2.5 Pro still have significant room for improvement. Abstract Powered by a large language model (LLM), a web browsing … view at source ↗
Figure 2
Figure 2. Distribution of websites and task types in WebChoreArena. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples in each task type in WebChoreArena. (i) Massive Memory tasks require accurately memorizing a large amount of information from the given page. (ii) Calculation tasks involve performing arithmetic operations. (iii) Long-Term Memory tasks require the agent to retain relevant information across many steps and interactions. (iv) Others involve tasks that require special or domain-specific operations. for functio… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison across different task types. This result reveals that the methodology of the agent itself has a substantial impact on its effectiveness across different task types. pre-training data of LLMs. We also attempted to run experiments using closed-source agents su…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 2 citations worldwide. Full citation record

  1. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.

Reference graph

Works this paper leans on

38 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google AI. Gemini 2.5: Our most intelligent ai model, 2025. URL https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking. Accessed: 2025-04-01

    work page 2025

  2. [2]

    Claude 3.7 sonnet system card

    Anthropic. Claude 3.7 sonnet system card. Technical report, Anthropic,

    work page

  3. [3]

    Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks

    Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault de Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. InNeurIPS, 2024

    work page 2024

  4. [4]

    Web agents with world models: Learning and leveraging environment dynamics in web navigation

    Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. InICLR, 2025

    2025

  5. [5]

    Spa-bench: A comprehensive benchmark for smartphone agent evaluation

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. InICLR, 2025

    work page 2025

  6. [6]

    The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467, 2024

    De Chezelles, Thibault Le Sellier, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, et al. The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467, 2024

    arXiv 2024

  7. [7]

    Brandon Chiou, Mason Choey, Mingkai Deng, Jinyu Hou, Jackie Wang, Ariel Wu, Frank Xu, Zhiting Hu, Hongxia Jin, Li Erran Li, Graham Neubig, Yilin Shen, and Eric P. Xing. Reasoneragent: A fully open source, ready-to-run agent that does research in a web browser and answers your queries, February 2025. URL https://reasoner-agent.maitrix.org/

    work page 2025

  8. [8]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. InNeurIPS, 2023

    work page 2023

Show all 38 references
  1. [9]

    Workarena: How capable are web agents at solving common knowledge work tasks? InICLR, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? InICLR, 2024

    2024

  2. [10]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. InACL, 2024

    2024

  3. [11]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  4. [12]

    Jace ai, 2025

    Jace AI. Jace ai, 2025. URLhttps://jace.ai/. Accessed: 2025-03-14

    2025

  5. [13]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. InACL, 2024

    2024

  6. [14]

    Tree search for language model agents.arXiv preprint arXiv:2407.01476, 2024

    Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents.arXiv preprint arXiv:2407.01476, 2024

    2024

  7. [15]

    Reinforcement learning on web interfaces using workflow-guided exploration

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. InICLR, 2018

    2018

  8. [16]

    Visualagentbench: Towards large multimodal models as visual foundation agents.arXiv preprint arXiv:2408.06327, 2024

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents.arXiv preprint arXiv:2408.06327, 2024

    2024 arXiv

  9. [17]

    Towards enterprise-ready computer using generalist agent.arXiv preprint arXiv:2503.01861, 2025

    Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Aviad Sela, Asaf Adi, and Nir Mashkif. Towards enterprise-ready computer using generalist agent.arXiv preprint arXiv:2503.01861, 2025. 21

    2025 arXiv

  10. [18]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InICLR, 2024

    2024

  11. [19]

    Browser use: Enable ai to control your browser, 2024

    Magnus Müller and Gregor Žuniˇc. Browser use: Enable ai to control your browser, 2024. URL https://github.com/browser-use/browser-use. Accessed: 2025-04-01

    2024

  12. [20]

    Gui agents: A survey.arXiv preprint arXiv:2412.13501, 2024

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey.arXiv preprint arXiv:2412.13501, 2024

    2024

  13. [21]

    Computer-using agent: Introducing a universal interface for ai to interact with the digi- tal world

    OpenAI. Computer-using agent: Introducing a universal interface for ai to interact with the digi- tal world. 2025. URL https://openai.com/index/computer-using-agent . Accessed: 2025-03-14

    2025

  14. [22]

    Webcanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373, 2024

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373, 2024

    2024 arXiv

  15. [23]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. InICLR, 2025

    2025

  16. [24]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. InICLR, 2025

    2025

  17. [25]

    World of bits: An open-domain platform for web-based agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. InICML, 2017

    2017

  18. [26]

    Beyond browsing: Api-based web agents.arXiv preprint arXiv:2410.16464, 2024

    Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents.arXiv preprint arXiv:2410.16464, 2024

    2024 arXiv

  19. [27]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. InICML, 2025

    2025

  20. [28]

    Webarena Leaderboard

    Webarena Team. Webarena Leaderboard. https://docs.google.com/spreadsheets/d/ 1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ. Accessed: 2025-04-01

    2025

  21. [29]

    Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  22. [30]

    Webwalker: Benchmarking llms in web traversal.arXiv preprint arXiv:2501.07572, 2025

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang. Webwalker: Benchmarking llms in web traversal.arXiv preprint arXiv:2501.07572, 2025

    2025 arXiv

  23. [31]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InNeurIPS, 2024

    2024

  24. [32]

    An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025

    2025

  25. [33]

    Agentoccam: A simple yet strong baseline for llm-based web agents

    Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm-based web agents. In ICLR, 2025

    2025

  26. [34]

    Assistantbench: Can web agents solve realistic and time-consuming tasks?arXiv preprint arXiv:2407.15711, 2024

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?arXiv preprint arXiv:2407.15711, 2024

    2024 arXiv

  27. [35]

    Mmina: Benchmarking multihop multimodal internet agents.arXiv preprint arXiv:2404.09992, 2024

    Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents.arXiv preprint arXiv:2404.09992, 2024. 22

    2024 arXiv

  28. [36]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. InICLR, 2024

    2024

  29. [37]

    Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models

    Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models. InACL, 2024. 23

    2024

  30. [2024]

    Accessed: 2025-03-14

    URL https://assets.anthropic.com/m/785e231869ea8b3b/original/ claude-3-7-sonnet-system-card.pdf. Accessed: 2025-03-14

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.