Pith. sign in

REVIEW 4 major objections 4 minor 38 references

WebArXiv: Evaluating Multimodal Agents on Time-Invariant arXiv Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WebArXiv proposes a time-invariant benchmark of 275 arXiv tasks with fixed snapshots and deterministic answers, plus a dynamic reflection mechanism that improves top agents by up to 8.9 points.

desk verdict A useful benchmark idea whose core reproducibility premise is asserted but never demonstrated, and the result tables contain errors that need correction. read the letter →

arxiv 2507.00938 v2 pith:EY6T3NRC submitted 2025-07-01 cs.IR cs.AIcs.DB

Zihao Sun , Ling Chen This is my paper
classification cs.IRcs.AIcs.DB
keywords webagentsmultimodalbenchmarkarXivtime-invarianttasksdynamicreflectionnavigationdeterministicgroundtruth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that web-agent evaluation can be made reproducible by grounding tasks in fixed snapshots of a stable site, arXiv, with exact answers that machines can verify without manual inspection. It offers 275 such tasks across five categories—site information, rules and accounts, paper retrieval, advanced search and filtering, and deep paper content extraction—together with standardized prompts, reference action trajectories, and deterministic ground truths. The same study identifies a common failure mode it calls Rigid History Reflection, where agents treat their most recent interaction steps as equally relevant, and proposes a lightweight dynamic reflection mechanism that selects the most useful prior step before each action. Across ten multimodal web agents, the reflection variant consistently improves on the base model, with the largest gains on GPT-o1 (from 56.7% to 61.8%) and Gemini-2.5 (from 51.1% to 60.0%). If the time-invariance premise holds, the field gains a stable measuring stick for navigation agents and a simple prompting mechanism that improves them.

What carries the argument

The central object is WebArXiv, a static, time-invariant benchmark of 275 arXiv tasks divided into five temporally stable categories, each with a frozen snapshot context, standardized prompts, reference trajectories, and deterministic ground truths. It carries the argument by removing answer drift as a confounding factor, so differences in success rates can be attributed to agent behavior rather than changing web content. The second mechanism is dynamic reflection, a prompting procedure that, at every interaction step, asks the model to choose the most relevant of the last three screenshots, then combines that chosen screenshot with the current view to generate the next action; this replaces the uniform last-three-steps memory that most agents use.

What would settle it

Reopen a subset of tasks after a documented arXiv help-page or interface change, for example re-asking 'How can I package my submission files?' on the current help page; if the live instruction differs from the frozen gold answer 'Create tar.gz and zip Files', the time-invariance premise is violated.

Watch

Extended reading notes

Core claim

The paper claims that a benchmark can be made time-invariant by anchoring each task to fixed arXiv webpage content, manually verifying gold answers for uniqueness and stability, and enforcing strict machine-checkable matching so that task success is unaffected by web drift or API changes. The second claim is that most web agents fail not from lack of context but from rigid history use: they keep a fixed number of recent interaction steps and weigh them all equally, which causes loops and missed information on dense pages. Dynamic reflection addresses this by asking the model to identify the single most relevant of the last three visual observations and combining that chosen observation with the current view before generating the next action. The evidence is a ten-agent evaluation, each run three times per task, showing that dynamic reflection beats uniform three-step memory and single-step memory, and lifts top models such as Gemini-2.5 from 51.1% to 60.0% and GPT-o1 from 56.7% to 61.8%.

Load-bearing premise

The load-bearing premise is temporal invariance: gold answers are verified once at construction time, and the paper does not specify how snapshots are hosted, refreshed, or re-checked, so if arXiv content drifts or the frozen snapshots diverge from the described interface, the deterministic ground truths go stale and the reproducibility claim breaks.

Editorial extensions

If this is right

  • WebArXiv can be re-run on new models as they ship without re-labeling, since answers are designed to be deterministic and time-invariant.
  • The dynamic reflection prompt is a zero-training intervention that lifts success rates for both frontier and smaller models, making it a cheap component to add to existing agent stacks.
  • The gap between categories, with Advanced Search and Filtering staying below 50 percent for almost all agents, points navigation research toward dense, form-heavy interfaces.
  • Overall scores separate agents clearly enough to rank them, giving the community a usable baseline for tracking progress over time.
  • Standardized prompts, reference trajectories, and strict matching make the benchmark usable for reinforcement learning training, not just evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The strongest test of the time-invariance premise would be to re-run the same 275 tasks after a known arXiv layout or help-page change and show that success rates are unchanged.
  • Dynamic reflection is a prompt-time intervention, so it should transfer to other static-knowledge domains, but the paper only demonstrates it on arXiv.
  • Because all tasks use arXiv's English interface, the benchmark measures English-only navigation; multilingual help pages may present different navigation behaviors and are an untested extension.
  • If the frozen snapshots are publicly accessible, a later agent could shortcut the navigation by memorizing the answer-bearing page, so an extension would be to rotate or randomize tasks to prevent memorization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces WebArXiv, a benchmark of 275 arXiv-based tasks across five categories of 55 tasks each, and claims that it provides reproducible and reliable evaluation by anchoring tasks in fixed web snapshots with deterministic ground truths and standardized action trajectories. It also proposes a lightweight dynamic reflection mechanism that selects the most relevant of the last three interaction steps before each action, and evaluates ten multimodal web agents (GPT-4o, GPT-o1, Gemini variants, SeeAct, LiteWebAgent, OpenWebAgent, etc.) on the benchmark. Results are reported as category-wise and overall task success rates, with an ablation comparing dynamic reflection to fixed-history baselines.

Significance. If properly substantiated, WebArXiv would be a useful stable evaluation instrument for web agents, and the dynamic reflection strategy is a simple, parameter-light intervention with potential to transfer to other navigation benchmarks. The paper's strengths include a concrete task construction pipeline, manual verification by three annotators, a broad comparison across ten models, and an open-source code release. However, the manuscript does not yet demonstrate the load-bearing reproducibility premise (snapshot availability, versioning, and refresh protocol), and it contains internal inconsistencies in the reported numbers and evaluation protocol that must be resolved before the headline claims can be accepted.

major comments (4)
  1. [Section 3.1 and Abstract] The abstract and contribution list claim that WebArXiv is anchored in fixed web snapshots, but the paper never specifies how snapshots are captured, stored, versioned, or served. No hashes, Memento/WARC endpoints, or replay harness are provided, and the anonymous link in the abstract is described only as 'code.' Without a retrievable frozen environment, the deterministic ground truths and standardized action trajectories cannot be independently checked, and the 'time-invariant' premise reduces to an assertion that live arXiv pages happen not to have changed after verification. Please describe the snapshot artifacts (or a restore/replay mechanism) and provide a re-verification or refresh protocol.
  2. [Tables 1 and 2] Several reported totals are inconsistent with the category values under the stated design of 55 tasks per category. For example, the GPT-4o + dynamic reflection row lists category values 63.6%, 60.0%, 38.2%, 34.5%, and 52.7%, which average to 49.8%, not the reported total of 38.4%. Similarly, GPT-o1's total of 56.7% exceeds its category average of 55.2%, and LiteWebAgent's total of 44.0% exceeds its category average of 42.4%. Please correct the tables or explain why the totals are weighted differently; as printed, these errors undermine confidence in all reported comparisons.
  3. [Sections 1, 3.2, and 4.1] Section 1 states that all answers are machine-verifiable, 'eliminating the need for manual inspection,' yet Section 3.2 describes three annotators reviewing each agent's full trajectory and assigning Correct/Incorrect/Partial-Correct labels. Section 4.1 then says success is determined by strict matching to gold-standard answers. Please clarify whether the reported success rates come from automatic string matching or from human annotation, and what role the Partial-Correct label and human review play in the reported numbers. If both are used, the protocol and the claimed 'no manual inspection' property need to be reconciled.
  4. [Section 4.3 and Table 3] The ablation compares dynamic reflection against the 'last 3 steps' baseline and concludes that the mechanism is effective, but Table 3 also includes a 'last step only' baseline. For GPT-4-Turbo, dynamic reflection achieves 40.2%, which is lower than the 43.6% success of simply using the last step. For GPT-o1, the improvement over last-step-only is only 1.8 points (61.8% vs 60.0%). With three runs per task and no variance or significance testing, these differences are not sufficient to support the claim that dynamic reflection is uniformly superior. Please report per-run results and statistical comparisons, and discuss why dynamic reflection underperforms the last-step-only baseline for GPT-4-Turbo.
minor comments (4)
  1. [Throughout] The manuscript contains multiple typos and grammatical errors: 'to to improve' (Contributions bullet 2), 'webagents' and 'most webagents handles' (Section 3.3), 'provides well-aligned' (Section 1), 'failed is on the right track' (Section 3.2), and inconsistent capitalization 'WebArxiv' vs 'WebArXiv'.
  2. [Table 3] The arrow notation in the header 'Successful (↑) Partial (↓) Failed (↓)' is never defined; please clarify what the arrows denote.
  3. [Figures] Figure 7 appears without an in-text citation, and several example figures (Figures 2-6) are not referenced in the body text; all figures should be cited in order.
  4. [References] The reference list contains a duplicated OpenWebAgent entry (the same Iong et al. 2024 work is listed twice) and inconsistent formatting for the Gemini 1.5 series, which appears under two different technical-report titles.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the benchmark is constructed from expert-generated tasks with manual verification, and the reflection method is evaluated as an in-domain comparison, not as a fitted prediction.

full rationale

The paper contains no derivation chain that reduces to its own inputs. WebArXiv is a newly constructed benchmark: tasks are drafted by human experts, filtered by semantic similarity, and answers are manually verified by three annotators. The dynamic reflection mechanism is a prompting strategy evaluated against base agents on the same benchmark, which is standard empirical evaluation rather than circular reasoning. The abstract claims tasks are anchored in fixed web snapshots, but the body provides no snapshot hosting, versioning, hashing, or replay protocol; this is a missing-evidence / external-validity problem, not a circularity, because no result is forced by definition or by a fitted parameter. There are no fitted parameters, no self-citations used as load-bearing support, and no known result renamed as a new contribution. The main limitations are the lack of external benchmark validation and the unverified temporal-invariance guarantee, both of which fall outside the circularity criteria.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two premises the paper does not fully demonstrate: temporal invariance of the tasks and their gold answers, and the sufficiency of three averaged runs without significance testing. Construction choices such as the 55-task cutoff and the semantic similarity threshold are unreported design decisions. No new physical or conceptual entities are introduced; 'Rigid History Reflection' is a named failure mode, not an entity.

free parameters (3)
  • History window size k = 3 = 3
    Baseline agents retain the last three interaction steps; Table 3 ablates k=1,2,3 but the reflection design and comparisons are anchored to this hand-chosen default.
  • Tasks per category = 55
    Filtering from 100 drafted candidates per category to 55 retained tasks; the cutoff is a construction choice not derived from data.
  • Semantic similarity filtering threshold = not reported
    Used with all-mpnet-base-v2 to remove redundant tasks in Section 3.1; the threshold value is not stated, so selection is not fully reproducible.
assumptions (4)
  • domain assumption arXiv task content is time-invariant and stable over time.
    Core premise of the benchmark (abstract, Section 3). arXiv metadata and help pages can change; the paper asserts stability but does not document re-verification dates.
  • domain assumption Gold-standard answers are unique, precise, and machine-verifiable.
    Section 3.2 and the experiment setup rely on deterministic matching, yet Section 3.2 also describes human annotators making Correct/Incorrect/Partial judgments, a tension the paper does not resolve.
  • ad hoc to paper Selecting the most relevant of the last three steps improves next-action decisions.
    Assumption underlying the dynamic reflection mechanism (Section 3.3, Figure 9); no mechanism-level analysis is provided.
  • domain assumption Three repeated runs with averaged success rate are sufficient to compare agents.
    Section 4.1; no variance, confidence intervals, or significance tests are reported, so differences of a few points may be noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WebArXiv: Evaluating Multimodal Agents on Time-Invariant arXiv Tasks." pith.science (2026). https://pith.science/paper/EY6T3NRC

@misc{pith2026250700938,
  author       = {Pith},
  title        = {Pith review of: WebArXiv: Evaluating Multimodal Agents on Time-Invariant arXiv Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EY6T3NRC}},
  note         = {Machine review of arXiv:2507.00938}
}
read the original abstract

Recent progress in large language models (LLMs) has enabled the development of autonomous web agents capable of navigating and interacting with real websites. However, evaluating such agents remains challenging due to the instability and inconsistency of existing benchmarks, which often rely on dynamic content or oversimplified simulations. In this work, we introduce WebArXiv, a static and time-invariant benchmark comprising 275 web-based tasks grounded in the arXiv platform. WebArXiv ensures reproducible and reliable evaluation by anchoring tasks in fixed web snapshots with deterministic ground truths and standardized action trajectories. Through behavioral analysis, we identify a common failure mode, Rigid History Reflection, where agents over-rely on fixed interaction histories. To address this, we propose a lightweight dynamic reflection mechanism that allows agents to selectively retrieve relevant past steps during decision-making. We evaluate ten state-of-the-art web agents on WebArXiv. Results demonstrate clear performance differences across agents and validate the effectiveness of our proposed reflection strategy.

Figures

Figures reproduced from arXiv: 2507.00938 by the authors.

Figure 1
Figure 1. WebArXiv task benchmark creation pipeline, illustrating the stages of task generation, LLM filtering, and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An organizational information retrieval case for arXiv. Given the task: “On arXiv’s About page, find the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. A user account management task on arXiv. Given the task: “How can I package my submission files?” [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: A paper discovery task on arXiv. Given the task: “State the name of the second Author of this paper: 3D [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: A search interaction task on arXiv’s advanced search page. Given the task: “Tell me how to search within [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: A publication detail retrieval task on arXiv. Given the task: “Provide the name of the university publishing [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Stacked bar chart of GPT-o1, showing task [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 9
Figure 9. Figure 9: Pseudocode for dynamic reflective agent across the last 3 steps in WebArXiv. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 19 canonical work pages

  1. [1]

    Hyungjoo Chae, Namyoung Kim, Minju Gwak, Gwanwoo Song, Jihoon Kim, Kai Ong, Seonghwan Kim, Dongha Lee, and Jinyoung Yeo. 2024. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In NeurIPS Workshop on System-2 Reasoning at Scale

    work page 2024

  2. [2]

    Google DeepMind. 2024. https://deepmind.google/technologies/gemini/ Gemini 1.5: Technical overview

    work page 2024

  3. [3]

    Google DeepMind. 2025. Gemini 1.5 and gemini flash: Multimodal models with 1 million token context. https://deepmind.google/technologies/gemini. Accessed: May 2025

    work page 2025

  4. [4]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. https://arxiv.org/abs/2306.06070. ArXiv:2306.06070

    arXiv 2023

  5. [5]

    Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2025. https://arxiv.org/abs/2503.09572 Plan-and-act: Improving planning of agents for long-horizon tasks . arXiv preprint arXiv:2503.09572

    arXiv 2025

  6. [6]

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2024. https://arxiv.org/abs/2305.11854 Multimodal web navigation with instruction-finetuned foundation models . In International Conference on Learning Representations (ICLR)

    arXiv 2024

  7. [7]

    Siddhant Garg, Harshita Bansal, Yihan Wang, Daniel Khashabi, and Ashish Sabharwal. 2025. Real: Benchmarking autonomous agents on deterministic simulations of real websites. arXiv preprint arXiv:2504.11543

    arXiv 2025

  8. [8]

    Petko Georgiev, Rohan Anil, and et al. 2023. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models . arXiv preprint arXiv:2312.11805

    arXiv 2023

Show all 38 references
  1. [9]

    Significant Gravitas. 2023. Auto-gpt: Self-improving ai agent using gpt-4. GitHub repository. https://github.com/Torantulino/Auto-GPT

    2023

  2. [10]

    Hongjin He, Ning Ding, Xu Han, Zhiyuan Liu, Rui Jiang, Jiawei Yan, and Maosong Sun. 2024 a . Pasa: A paper searching agent with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  3. [11]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024 b . https://arxiv.org/abs/2401.13919 Webvoyager: Building an end-to-end web agent with large multimodal models . arXiv preprint arXiv:2401.13919

    2024 arXiv

  4. [12]

    Iat Long Iong, Xiao Liu, Yuxuan Chen, Hanyu Lai, Shuntian Yao, Pengbo Shen, Hao Yu, Yuxiao Dong, and Jie Tang. 2024. https://aclanthology.org/2024.acl-demos.8/ Openwebagent: An open toolkit to enable web agents . In ACL Demo Track

    2024

  5. [13]

    Iong, Iat Long and Liu, Xiao and Chen, Yuxuan and Lai, Hanyu and Yao, Shuntian and Shen, Pengbo and Yu, Hao and Dong, Yuxiao and Tang, Jie . 2024. Openwebagent: An open toolkit to enable web agents on large language models. In ACL 2024 System Demonstration Track

    2024

  6. [14]

    Jungwoo Kim, Minsang Kim, and Sungjin Lee. 2025. https://arxiv.org/abs/2502.04774 Sedi-instruct: Enhancing alignment of language models through self-directed instruction generation . arXiv preprint arXiv:2502.04774

    2025 arXiv

  7. [15]

    Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste

    Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan...

    2024

  8. [16]

    Linhao Luo, Zicheng Zhao, Gholamreza Haffari, Dinh Phung, Chen Gong, and Shirui Pan. 2025. https://arxiv.org/abs/2502.01113 Gfm-rag: Graph foundation model for retrieval augmented generation . arXiv preprint arXiv:2502.01113

    2025

  9. [17]

    Reiichiro Nakano and 1 others. 2023. Webgpt: Browser-assisted question-answering with human feedback. In International Conference on Learning Representations (ICLR)

    2023

  10. [18]

    OpenAI. 2023 a . Gpt-4 technical report. https://arxiv.org/abs/2303.08774. ArXiv:2303.08774

    2023 arXiv

  11. [19]

    OpenAI. 2023 b . https://openai.com/gpt-4 Gpt-4 turbo overview

    2023

  12. [20]

    OpenAI. 2024 a . Gpt-4o technical report. https://openai.com/index/gpt-4o. Accessed: May 2024

    2024

  13. [21]

    OpenAI. 2024 b . https://openai.com/index/gpt-4o Gpt-4o technical report

    2024

  14. [22]

    Yuxiang Pan, Difei Kong, Shuyan Zhou, Chuan Cui, Yizhou Leng, Bing Jiang, Haoran Liu, Yujie Shang, Shuchang Zhou, Tong Wu, and Zhaojun Wu. 2024. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373

    2024 arXiv

  15. [23]

    Timo Schick and 1 others. 2024. Toolformer: Language models can teach themselves to use tools. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)

    2024

  16. [24]

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. 2023. https://arxiv.org/abs/2306.00245 From pixels to ui actions: Learning to follow instructions via graphical user interfaces . arXiv...

    2023 arXiv

  17. [25]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://arxiv.org/abs/2303.11366 Reflexion: Language agents with verbal reinforcement learning . In Advances in Neural Information Processing Systems (NeurIPS)

    2023 arXiv

  18. [26]

    Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, and Mohit Iyyer. 2025. Bearcubs: A benchmark for computer-using web agents. In Proceedings of the 2025 Conference on Web Intelligence and Autonomous Systems

    2025

  19. [27]

    Hugo Touvron. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  20. [28]

    Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, and Manuela Veloso. 2024. Adapting multimodal web agents with few-shot learning from human demonstrations. In Proceedings of the 2024 Conference on Neural Information Processing Systems (NeurIPS)

    2024

  21. [29]

    Shunyu Yao and 1 others. 2023. https://arxiv.org/abs/2210.03629 React: Synergizing reasoning and acting in language models . arXiv preprint arXiv:2210.03629

    2023 arXiv

  22. [30]

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of llm-based agents. arXiv preprint arXiv:2503.16416

    2025 arXiv

  23. [31]

    Ori Yoran, Shoval J Amouyal, Chaitanya Malaviya, Ben Bogin, Omer Press, and Jonathan Berant. 2024. Assistantbench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407.15711

    2024 arXiv

  24. [32]

    Danqing Zhang, Balaji Rama, Jingyi Ni, Shiying He, Fu Zhao, Kunyu Chen, Arnold Chen, and Junyu Cao. 2025. Litewebagent: The open-source suite for vlm-based web-agent applications. https://arxiv.org/abs/2503.02950. ArXiv:2503.02950

    2025 arXiv

  25. [33]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2023. Seeact: A multi-modal agent for web navigation with visual perception and action. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2023

  26. [34]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. https://arxiv.org/abs/2401.01614 Gpt-4v(ision) is a generalist web agent, if grounded . arXiv preprint arXiv:2401.01614

    2024 arXiv

  27. [35]

    Xu, Haozhe Li, Hang Lv, Amanpreet Singh, Alexander Ratner, Anca Dragan, and Chelsea Finn

    Shuyan Zhou, Frank F. Xu, Haozhe Li, Hang Lv, Amanpreet Singh, Alexander Ratner, Anca Dragan, and Chelsea Finn. 2024. Webarena: A realistic web environment for building autonomous agents. In Proceedings of the 12th International Conference on Learning Representations (ICLR)

    2024

  28. [36]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and 1 others

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. https://arxiv.org/abs/2307.13854. ArXiv:2307.13854

    2023 arXiv

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.