Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

Coding Agents with Multimodal Browsing are Generalist Problem Solvers

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A coding agent augmented with multimodal browsing and web search matches or beats specialized agents across three diverse benchmarks.

desk verdict Useful cross-benchmark agent study with honest same-LLM controls, but the headline SOTA claims are confounded by model choice and validation-tuned search API. read the letter →

arxiv 2506.03011 v1 pith:XOUDILWS submitted 2025-06-03 cs.CL

Aditya Bharat Soni , Boxuan Li , Xingyao Wang , Valerie Chen , Graham Neubig This is my paper
classification cs.CL
keywords generalistagentmultimodalwebbrowsingSet-of-MarkspromptingsearchAPIcodeexecutionbenchmarkgeneralizationtooluse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that specialization is not necessary: one agent built on a small set of general tools can match or beat agents hand-tuned for software engineering, general web assistance, and digital coworker tasks. The agent, OpenHands-Versa, starts from a coding agent and adds multimodal web browsing (screenshots with overlaid action labels), a web search API, and Markdown-based viewing of multimodal files. On three benchmarks — GAIA, SWE-Bench Multimodal, and The Agent Company — it reports state-of-the-art or competitive resolve rates with the same configuration across all three. If true, the result suggests that tool breadth plus a strong backbone LLM can substitute for benchmark-specific architecture and prompt engineering.

What carries the argument

The mechanism that carries the result is the visual browsing observation. Instead of feeding the model only the accessibility tree, OpenHands-Versa annotates a viewport screenshot with bounding boxes and alphanumeric labels on every interactable element (Set-of-Marks), and combines this with the accessibility tree for off-screen context. A browsing condenser keeps only the most recent browsing observation, preventing context bloat and cost growth. Around this, a web-search API tool and Markdown-converted file viewing give the agent the information-access abilities, while the unchanged shell and IPython tools provide code execution and editing. The design makes no benchmark-specific choices; the same toolset and prompts are used everywhere.

What would settle it

Run the strongest baselines on all three benchmarks using the same backbone language model that OpenHands-Versa uses; if their scores rise to or above the reported numbers, the toolset is not the cause of the margin. Separately, remove the visual browser and search API from OpenHands-Versa while keeping the same model — the paper's own OpenHands comparison predicts the scores should drop to roughly OpenHands levels on GAIA and The Agent Company.

Watch

Extended reading notes

Core claim

The discovery this paper puts forward is that a single agent need not be specialized to be competitive everywhere. OpenHands-Versa keeps the base coding agent's code editing and execution, and adds visual web browsing via Set-of-Marks screenshots, API-based web search, and Markdown-based multimodal file viewing. With the same configuration across all three benchmarks, it reports 51.16% resolve rate on GAIA, 34.43% resolve rate on SWE-Bench Multimodal, and 33.14% full completion score on The Agent Company, improving on the best reproducible published results by 1.3, 9.1, and 9.1 absolute points. The authors read these results as evidence that tool breadth plus a strong backbone model, without benchmark-specific prompts, planners, or linters, is enough for generalist problem solving.

Load-bearing premise

The reported superiority over prior agents assumes that differences in success rates come from tool and agent design rather than from the different backbone language models used for baselines.

Editorial extensions

If this is right

  • If the result holds, benchmark-specific agent engineering is largely unnecessary; adding general-purpose perception and search tools to a strong coding agent gives broad coverage.
  • Task families that previously required separate agents could be served by one agent, simplifying deployment: one runtime, one set of credentials, and one evaluation harness.
  • The reported failure of leading multi-agent systems to generalize beyond their target benchmark suggests that narrow tooling, rather than the backbone model alone, is often the bottleneck.
  • Identifiable failure modes — over-reliance on search snippets, CAPTCHA-blocked sites, and weak self-testing in coding — become tractable targets for improvement without redesigning the architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper reports that swapping the search API alone moves GAIA validation resolve rate from 56.96% (Brave) to 64.24% (Tavily), the search provider is likely a larger driver of the GAIA margin than the agent architecture; a head-to-head ablation across providers would test this.
  • A matched-model rerun of the best baselines (using the same backbone as OpenHands-Versa) would separate tool-breadth effects from raw model capability; the paper does not presently control for this.
  • The same 'coding plus visual browsing plus search plus file access' recipe is a natural candidate for other long-horizon task families, such as desktop automation, where the bottleneck is again perception of visual state rather than planning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes OpenHands-Versa, a single-agent system built on the OpenHands framework by adding multimodal web browsing (Set-of-Marks screenshots plus accessibility-tree context), API-based web search (Tavily, with Exa and Brave also supported), multimodal file-to-Markdown conversion, and a periodic planning prompt. The agent is evaluated on GAIA, SWE-Bench Multimodal, and The Agent Company. The authors report state-of-the-art or competitive results on all three benchmarks (51.16% on GAIA, 34.43% on SWE-Bench M, and 33.14% full-completion on The Agent Company with claude-sonnet-4), and they include a same-LLM comparison against OpenHands v0.28.1 with claude-3.7 showing large gains on GAIA and The Agent Company and a 0.39-point decrease on SWE-Bench M. The paper also analyzes tool-use distributions, gives error analyses, and studies the effect of search API choice on the GAIA validation split.

Significance. The central hypothesis—that a modest set of general tools, rather than domain-specific machinery, can let a single agent perform well across coding, general assistance, and digital-worker benchmarks—is timely and plausible. The paper's strengths include an open-source implementation, a matched-LLM comparison against the base OpenHands agent, and a transparent limitations section. If the headline claims were fully controlled, this would be a useful baseline for generalist agent research. However, the strongest cross-system claims currently rest on comparisons that mix backbone LLMs and on a validation-selected search API, so the significance depends on additional controls. The 9.09-point SWE-Bench M gain and the 1.33-point GAIA margin are not yet attributable to the toolset rather than to model choice or benchmark-specific tuning.

major comments (5)
  1. [§3.2, Table 2] The headline comparisons to prior state-of-the-art systems are confounded by backbone LLM. Versa's 34.43% on SWE-Bench M uses claude-sonnet-4, while Agentless-Lite's 25.34% uses claude-3.5-sonnet, and Versa's GAIA 51.16% uses claude-3.7 or claude-sonnet-4 while OpenDeepResearch's 49.83% uses o1. The same table's matched control shows Versa with claude-3.7 at 31.33% versus OpenHands v0.28.1 at 31.72% on SWE-Bench M, so the tool additions do not improve coding beyond the base agent and the 9.09-point margin over Agentless-Lite may be entirely model-driven. Please rerun the strongest baselines under matched backbone conditions, or explicitly restrict the state-of-the-art claims to matched comparisons.
  2. [§5.3, Table 2] The GAIA test result is not robustly separated from benchmark-specific tuning. Section 5.3 reports that switching the search API on the GAIA validation split changes resolve rate from 56.96% (Brave) to 64.24% (Tavily), and Tavily is then used for the reported test scores; with roughly 40% of GAIA tool calls being search queries, this validation-based selection can inflate the test result. In addition, all numbers come from single runs with temperature 0 but no repeated trials, so the 1.33-point GAIA margin over OpenDeepResearch—likely a handful of tasks—has no quantified uncertainty. Please report repeated runs with variance, and treat the validation-based API choice as a tuned component in the conclusions.
  3. [§2.2, Appendix B] The paper's motivating question asks for the minimal set of general tools, but no experiment isolates the contribution of any single addition. The only architectural comparison is the full OpenHands-Versa versus OpenHands v0.28.1, which jointly changes visual browsing, search API, multimodal file viewing, and planning; the hyperparameters k=1 and tau=10 are fixed without sensitivity analysis. The 'minimal set' conclusion is therefore not directly tested. An ablation that removes each tool, or at least a sensitivity study over k and tau, is needed to support the minimality claim; otherwise the conclusion should be limited to 'a small set of tools is sufficient.'
  4. [§4, Table 2] The reported The Agent Company improvement over the best baseline is numerically inconsistent. Section 4 states an improvement of 6.9 points full and 6.8 points partial over the best-performing baseline. From Table 2, the best prior baseline is OpenHands v0.14.2 with claude-3.5 sonnet (24.00% full, 34.40% partial); Versa with claude-3.7 gives 30.86%/40.18% (6.86/5.78 point gains) and with claude-sonnet-4 gives 33.14%/43.19% (9.14/8.79 point gains). Neither row yields 6.8 partial. Please correct the numbers and state which model row is being compared; this matters because the abstract's 9.1-point improvement refers to the claude-sonnet-4 row, not the matched claude-3.7 row used elsewhere in the discussion.
  5. [§4, §E] The claim that existing state-of-the-art multi-agent systems fail to generalize is under-supported: only OWL-roleplaying is actually evaluated outside its target domain (on The Agent Company), while Magentic-One and OpenDeepResearch are not run on the other benchmarks. Appendix E acknowledges the cost-based limitation, but the abstract and conclusion nonetheless assert the generalization failure as an empirical finding. Please either provide matched cross-benchmark evaluations of at least one representative specialist baseline per benchmark, or revise the claim to state that the capability comparison in Table 1 is predictive rather than empirically demonstrated.
minor comments (5)
  1. [§3.3] The phrase 'pass @ 1' should be formatted as pass@1, and the definition for checkpoint-based The Agent Company metrics should be made explicit in the same paragraph.
  2. [§5.2, Table 3] The GAIA error analysis uses the validation split, while the SWE-Bench M and The Agent Company examples are described from test trajectories; this asymmetry should be stated in the table caption or surrounding text.
  3. [§2.2] The sentence 'OpenHands-Versa also supports the use of Exa and Brave APIs' is not accompanied by results for those APIs on the test splits; clarifying that Exa and Brave were only compared on GAIA validation would avoid ambiguity.
  4. [Figure 2] The legend labels in the tool-use distribution figure are difficult to read at print size, and the caption does not identify which tools are newly added or modified; a short list in the caption would help.
  5. [Abstract, §4] The abstract reports 9.1-point gains on both SWE-Bench M and The Agent Company, while Section 4 reports a 6.9-point gain on The Agent Company; after reconciling with Table 2 these refer to different backbone rows, but the manuscript should state this explicitly to avoid apparent contradiction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GAIA, SWE-Bench M, and The Agent Company results are held-out external measurements, and the Tavily choice is validation tuning rather than a definitional reduction.

full rationale

OpenHands-Versa is an empirical systems paper with no mathematical derivation chain, so the classic circularity patterns (definitional identity, fitted-parameter-as-prediction, self-citation uniqueness theorems) do not apply. The central claims—that a single agent with code editing and execution, web search, multimodal browsing, and file access reaches competitive resolve rates on GAIA, SWE-Bench M, and The Agent Company—are evaluated on held-out test splits against external benchmarks, and the reported numbers are independent measurements rather than consequences of the paper's definitions. The only place one might suspect circularity is the search-API choice in Section 5.3: the authors evaluate Brave, Exa, and Tavily on the GAIA validation split, observe Tavily at 64.24%, and then use Tavily in the main experiments. This is a standard hyperparameter/model-selection step, not a reduction by construction: the GAIA test result (51.16%) is a separate held-out measurement, and the paper does not present the validation scores as predictions of test scores. The self-citations to OpenHands, BrowserGym, and The Agent Company describe frameworks and benchmarks used as infrastructure, not load-bearing arguments, and no uniqueness theorem is imported to forbid alternative designs. The backbone-LLM mismatch between OpenHands-Versa and some baselines is a legitimate experimental confound and a correctness risk, but it does not make the result circular. Accordingly, there is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system is an empirical combination of existing components, so it introduces no new mathematical axioms. The load-bearing assumptions are all empirical comparability assumptions: that leaderboard results from different papers can be compared directly, that the three benchmarks capture generalist diversity, and that the same-LLM OpenHands comparison isolates the effect of the added tools. Two hand-set hyperparameters (k=1, tau=10) and the validation-set choice of Tavily as the search API are the only paper-level parameter choices; none is ablated.

free parameters (3)
  • browsing context window k = 1
    Set by hand in Appendix B; the agent retains only the most recent browsing observation. No sensitivity analysis is given, and this choice can affect browsing performance.
  • planning interval tau = 10
    Set by hand in Appendix B; a planning prompt is injected every 10 steps. No ablation is reported, so the contribution of this hyperparameter to the results is unknown.
  • search API choice = Tavily
    In Section 5.3, three search APIs are scored on the GAIA validation split and Tavily performs best; the reported GAIA test results use that selection. This is a configuration fitted to a validation set rather than a free numeric parameter.
assumptions (4)
  • domain assumption Comparability of leaderboard results across papers
    Table 2 takes success rates from different papers, computed with different backbone LLMs, evaluation harnesses, and compute budgets, as directly comparable evidence of agent design quality.
  • domain assumption Three benchmarks proxy generalist task diversity
    Section 3.1 asserts these benchmarks cover a diverse range of capabilities, but no formal argument or broader benchmark set supports the claim that a generalist for these three generalizes broadly.
  • domain assumption Same-LLM comparison to OpenHands isolates the effect of added tools
    Section 4 compares OpenHands-Versa to OpenHands v0.28.1 with the same claude-3.7-sonnet backbone and attributes differences to the new tools, assuming no other interaction effects.
  • domain assumption Backbone LLMs used are adequate and comparable enough
    The central SOTA comparisons assume claude-sonnet-4 or claude-3.7-sonnet differences do not dominate the small margins, e.g., the 1.33 point GAIA gain over OpenDeepResearch.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Coding Agents with Multimodal Browsing are Generalist Problem Solvers." pith.science (2026). https://pith.science/paper/XOUDILWS

@misc{pith2026250603011,
  author       = {Pith},
  title        = {Pith review of: Coding Agents with Multimodal Browsing are Generalist Problem Solvers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOUDILWS}},
  note         = {Machine review of arXiv:2506.03011}
}
read the original abstract

Modern human labor is characterized by specialization; we train for years and develop particular tools that allow us to perform well across a variety of tasks. In addition, AI agents have been specialized for domains such as software engineering, web navigation, and workflow automation. However, this results in agents that are good for one thing but fail to generalize beyond their intended scope. One reason for this is that agent developers provide a highly specialized set of tools or make architectural decisions optimized for a specific use case or benchmark. In this work, we ask the question: what is the minimal set of general tools that can be used to achieve high performance across a diverse set of tasks? Our answer is OpenHands-Versa, a generalist agent built with a modest number of general tools: code editing and execution, web search, as well as multimodal web browsing and file access. Importantly, OpenHands-Versa demonstrates superior or competitive performance over leading specialized agents across three diverse and challenging benchmarks: SWE-Bench Multimodal, GAIA, and The Agent Company, outperforming the best-performing previously published results with absolute improvements in success rate of 9.1, 1.3, and 9.1 points respectively. Further, we show how existing state-of-the-art multi-agent systems fail to generalize beyond their target domains. These results demonstrate the feasibility of developing a generalist agent to solve diverse tasks and establish OpenHands-Versa as a strong baseline for future research.

Figures

Figures reproduced from arXiv: 2506.03011 by the authors.

Figure 1
Figure 1. Comparison of OpenHands-Versa with previously published SOTA agents and OpenHands [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. This figure presents some interesting insights into the behavior of OpenHands-Versa and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Example screenshot of a webpage with set-of-marks annotation [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM Agents Can See Code Repositories

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    Visual graphs of repository structure added to text inputs for multimodal LLM agents reduce token consumption by up to 26% while maintaining or improving issue-resolution accuracy.

  2. SERA: Soft-Verified Efficient Repository Agents

    cs.CL 2026-01 conditional novelty 6.0 of 10

    SERA trains competitive coding agents via supervised finetuning on synthetic trajectories verified only by patch-overlap recall, cutting data-generation cost by roughly two orders of magnitude.

  3. How can we assess human-agent interactions? Case studies in software agent design

    cs.AI 2025-10 conditional novelty 6.0 of 10

    PULSE combines sparse human ratings with prediction-powered inference to cut confidence intervals by ~40% and shows LLM choice matters more than scaffolding for user satisfaction.

  4. MetaAgent: Toward Self-Evolving Agent via Tool Meta-Learning

    cs.AI 2025-08 reject novelty 6.0 of 10

    The paper advertises a self-evolving tool-using agent that beats baselines, but the body contains a different math paper, leaving the performance claims unsupported.

Reference graph

Works this paper leans on

26 extracted references · 11 canonical work pages · cited by 4 Pith papers

  1. [1]

    The "think" tool: Enabling claude to stop and think in complex tool use situations

    Anthropic(2025). The "think" tool: Enabling claude to stop and think in complex tool use situations. A new tool that improves Claude’s complex problem-solving performance

    work page 2025

  2. [2]

    Tapeagents: a holistic framework for agent development and optimization.arXiv preprint arXiv:2412.08445, 2024

    Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, et al. Tapeagents: a holistic framework for agent development and optimization.arXiv preprint arXiv:2412.08445, 2024

    arXiv 2024

  3. [3]

    Xu, Siva Reddy, Gra- ham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados

    Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexan- dre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Gra- ham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados. The browsergym ecosys...

    work page 2025

  4. [4]

    Agentless-lite

    Dunn(2025). Agentless-lite

    work page 2025

  5. [6]

    Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    arXiv 2024

  6. [7]

    Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025

    Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025

    work page 2025

  7. [8]

    Brave search api

    Brave Software Inc. Brave search api

    work page

  8. [9]

    SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    2024

Show all 26 references
  1. [10]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  2. [11]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  3. [12]

    Browser use: Enable ai to control your browser, 2024

    Magnus Müller and Gregor Žuni ˇc. Browser use: Enable ai to control your browser, 2024

    2024

  4. [13]

    Open-source deepresearch – freeing our search agents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier. Open-source deepresearch – freeing our search agents, 2025

    2025

  5. [14]

    Tavily search api

    Tavily-AI. Tavily search api. 10

  6. [15]

    Hopper, and John Langford

    Luis von Ahn, Manuel Blum, Nicholas J. Hopper, and John Langford. Captcha: Using hard ai problems for security. InAdvances in Cryptology - EUROCRYPT 2003, International Conference on the Theory and Applications of Cryptographic Techniques, Warsaw, Poland, May 4-8, 2003, Procee...

    2003

  7. [16]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. InThe Thirteenth International Conference on Learning Representat...

    2024

  8. [17]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024

    2024 arXiv

  9. [18]

    Agentless: Demystifying llm-based software engineering agents.arXiv preprint, 2024

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents.arXiv preprint, 2024

    2024

  10. [19]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for op...

    2024

  11. [20]

    Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. T...

    2024

  12. [21]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    2023

  13. [22]

    SWE-agent: Agent-computer interfaces enable automated soft- ware engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated soft- ware engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  14. [23]

    SWE-bench multimodal: Do AI systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. SWE-bench multimodal: Do AI systems generalize to visual software domains? In The Thirte...

    2025

  15. [24]

    Agentoccam: A simple yet strong baseline for llm-based web agents.arXiv preprint arXiv:2410.13825, 2024

    Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm-based web agents.arXiv preprint arXiv:2410.13825, 2024

    2024 arXiv

  16. [25]

    Multi-swe-bench: A multilingual benchmark for issue resolving, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: A multilingual benchmark fo...

    2025

  17. [26]

    Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms.arXiv preprint arXiv:2502.07942, 2025

    Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Le- andro Z Agudelo, Peter Qian, and Tianlong Chen. Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms.arXiv preprint arXiv:2502.07942, 2025

    2025 arXiv

  18. [27]

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. 11 A Webpage Screenshot with Set-of-...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.