Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Top AI agent solves only 40% of interactive CAPTCHAs, humans 93%.

desk verdict Useful benchmark with a real gap to fill, but the headline human-vs-agent numbers rest on an undocumented human baseline and single-run agent evaluations. read the letter →

arxiv 2505.24878 v1 pith:JLIPHWG3 submitted 2025-05-30 cs.AI cs.CLcs.CVcs.LG

Yaxin Luo , Zhaoyi Li , Jiacheng Liu , Jiacheng Cui , Xiaohan Zhao , Zhiqiang Shen This is my paper
classification cs.AIcs.CLcs.CVcs.LG
keywords CAPTCHAbenchmarkmultimodalLLMagentswebautomationinteractivevisualreasoningdepthbrowser-usehuman-AIperformancegapagentevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that current multimodal LLM agents, despite strong performance on static visual tasks, largely fail at interactive CAPTCHAs: in a new benchmark of 225 puzzles across 20 types, the best agent (Openai-o3 over Browser Use) reaches only 40% pass@1 while humans score 93.3%. To make this measurable, it introduces Open CaptchaWorld, an open web-based platform where agents must perceive screenshots and perform multi-step clicks, drags, and key actions in a closed loop, and a task-agnostic metric, CAPTCHA Reasoning Depth, that counts the atomic reasoning and motor steps a puzzle requires. If correct, the result means CAPTCHAs remain a major unsolved blocker for real-world web automation, and current evaluations that filter CAPTCHAs out give an inflated sense of agent competence.

What carries the argument

The central object is the web-based benchmark platform Open CaptchaWorld: 20 curated CAPTCHA types (225 puzzles), each instantiated as an interactive web page on a local server where an agent receives screenshots and issues discrete actions until it submits. The argument is carried by two mechanisms: the closed-loop Browser-Use agent protocol that mediates perception, memory, goal-setting, and action, and the proposed metric CAPTCHA Reasoning Depth, $D(T)=\sum_i I[s_i \in S_T]$, defined as the number of atomic steps from a fixed checklist (visual, cognitive, motor, state-reveal) needed to solve puzzle $T$, estimated by human annotators and by the models themselves to expose the reasoning mismatch. Each puzzle type is instantiated with at least ten diverse variants, and the whole loop is modeled as a finite-horizon partially observable Markov decision process.

What would settle it

Run the same Browser-Use plus Openai-o3 agent, with the identical prompt and closed-loop protocol, against 100 real reCAPTCHA v2 image challenges on live pages; if its success rate approaches the human level while its Open CaptchaWorld rate stays near 40%, the benchmark does not predict real-world difficulty. Alternatively, a re-run of the human baseline with a pre-registered protocol and sample size that yields a success rate far below 93.3% would undercut the claimed human-agent gap.

Watch

Extended reading notes

Core claim

The discovery is a large and consistent performance gap: across nine state-of-the-art MLLM backbones in the Browser-Use agent, none exceeds 40% success on the benchmark, while human participants reach 93.3%. The paper argues this gap stems from a mismatch between agents' over-segmented, literal reasoning and humans' holistic, experience-compressed problem-solving, and that the benchmark's interactive puzzles, unlike static perception tasks, expose exactly this limitation. It further claims the CAPTCHA Reasoning Depth metric aligns with this behavioral difference: models assign higher depth scores than humans because they treat every sub-action as a distinct reasoning step.

Load-bearing premise

The benchmark's synthetic puzzles, designed and generated by humans and GPT-4o and served on a local server, are assumed to represent the interactive CAPTCHAs agents would meet on production websites, and the human baseline of 93.3% is reported without a documented sample or protocol.

Editorial extensions

If this is right

  • CAPTCHA-containing pages should not be filtered out of web-agent benchmarks; success on filtered benchmarks overstates deployable autonomy.
  • Certain puzzle families are unsolved by every tested model, including Slide Puzzle, Dice Count, Coordinates, Path Finder, Place Dot, Connect Icon, Click Order, Misleading Click, and Pick Area, isolating which abilities (spatial manipulation, counting, path reasoning, fine motor control) are missing.
  • Cost-performance data imply that closing the gap needs not only better perception but better reasoning efficiency: Openai-o1 spent $94.6 for only 5% success, while cheaper models like DeepSeek-V3 and Claude-3.5-Haiku offer better balance at low accuracy.
  • The open platform provides a reusable infrastructure for tracking progress as new MLLMs are released, with the leaderboard and type-by-type support matrix as diagnostics.
  • The CAPTCHA Reasoning Depth annotations give a task-level difficulty axis that can be used to stratify agent failures by cognitive complexity rather than by surface puzzle type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because all puzzles are synthetic and hosted locally, live production CAPTCHAs with anti-bot timing, audio variants, and biometric mouse-movement checks could plausibly be even harder, making the 40% figure an upper bound on real-world agent performance.
  • The observed overthinking pattern suggests a concrete design fix, such as an action budget or a hierarchical planner that suppresses redundant state-monitoring steps, which could be ablated on this platform.
  • Reasoning depth estimates from models could serve as a cheap proxy for expected difficulty and cost, enabling dynamic task selection or curriculum ordering in agent training.
  • A natural next test is to fine-tune a small policy on Open CaptchaWorld and measure transfer to live reCAPTCHA or hCaptcha; the benchmark's practical value largely stands or falls on that transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Open CaptchaWorld, a web-based benchmark and platform for evaluating multimodal LLM agents on interactive CAPTCHA puzzles. The benchmark spans 20 CAPTCHA types with 225 items and proposes a 'CAPTCHA Reasoning Depth' metric that counts atomic cognitive and motor steps. The authors evaluate several MLLM agents (GPT-4o, GPT-4.1, Claude-3.7-Sonnet, Gemini2.5-Pro, Openai-o3, etc.) using the Browser Use agent framework in a zero-shot setting. The headline claim is that state-of-the-art agents succeed on at most 40% of puzzles (Openai-o3), far below a human baseline of 93.3%, and the paper provides a failure-case taxonomy. The manuscript positions this as the first open-source CAPTCHA benchmark for interactive agent evaluation.

Significance. If the central empirical claim is verified, this benchmark fills a real gap: existing web-agent benchmarks deliberately filter out CAPTCHAs, while real-world deployment must contend with them. The resource itself—an open platform with 225 puzzles across 20 interactive types, human-annotated ground truth, and a reasoning-depth checklist—is potentially useful to the community for diagnosing agent limitations. The failure-case analysis (overthinking, over-segmentation, interface misunderstanding) is qualitatively informative. However, the significance of the headline 40% vs. 93.3% gap is currently undercut by missing experimental detail and internal inconsistencies in the reported tables, as detailed below.

major comments (3)
  1. [§4.2, Table 1] The human baseline of 93.3% is reported without protocol details: no number of participants, number of puzzles attempted per participant, selection criteria, time limits, or disclosure of whether the participants were the same annotators who designed the puzzles and ground truth (see §3.3, which first mentions the 93.3% figure in the data-curation context). Since the headline claim is the 53.3-point gap between agents and humans, this baseline must be documented and defended as a fair comparison point.
  2. [§4.2, Table 1 and §4.1] The agent success rates appear to be point estimates from a single run per model, and the text in §4.2 says 'agents go through all puzzle types exactly once,' implying 20 trials per model. Under that reading, Table 1 values such as GPT-4o's 5.7% cannot correspond to an integer number of successes out of 20, and no confidence intervals are provided. Section 4.1 is an empty heading with no actual setup text (no number of runs, temperature, max steps, seed, or browser version), making the results unreproducible. The empirical claim needs either the true trial counts per model or a corrected protocol description with variance estimates.
  3. [Table 2] Table 2's per-type support matrix is inconsistent with Table 1's pass@1 rates. For example, Openai-o1 shows checkmarks for 5 of 20 types but Table 1 reports 5.0% pass@1 (which would be 1 success if there are 20 puzzles); Claude-3.5-Haiku shows 4 checkmarks but 15% (3/20); DeepSeek-V3 shows 2 checkmarks but 20% (4/20); and Openai-o3 shows 7 checkmarks but 40% (8/20). These discrepancies need to be reconciled—either the success criterion in Table 2 is different (e.g., at least one success across multiple instances per type) or the trial counts are different—and the relationship between the tables must be stated explicitly.
minor comments (4)
  1. [§4.1 (heading)] The heading '4.1 Experimental Setup' is followed directly by '4.2 Success Rate of Multimodal Agents on Open CaptchaWorld' with no text in between; the reader is left without any description of the evaluation setup beyond the statement in §4.2. Please fill in this section or merge it into §4.2.
  2. [Related Work] The sentence 'ToolBench [15] isolates tool-use challenges' cites [15], which is VisualWebArena; the citation appears to be incorrect and should point to the actual ToolBench reference.
  3. [Figure 5 caption] The caption reads 'Open CaptchaWorld Date Curation Pipeline'; 'Date' should be 'Data'.
  4. [§3.2, Eq. (1)] The formula for D(T) is typeset as 'NX i=1' instead of a proper summation; please fix the mathematical formatting. Also, the definition refers to 'the minimal number of atomic steps' but the annotation procedure describes averaging observed steps across annotators; clarify whether D(T) is a post-hoc annotation average or a theoretical minimum.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: the agent success rates in Table 1 are independent closed-loop measurements, and the reasoning-depth metric is a descriptive annotation tool, not a fitted predictor. Only a non-load-bearing tautological phrase ('confirms ... cognitive difficulty') and an unreported human-baseline protocol remain.

  1. self definitional [Section 3.2 (CAPTCHA Reasoning Depth), paragraph following Fig. 3]
    "Across the dataset, we observe high structural diversity: the average reasoning depth per task is 2.94 with a standard deviation of 0.92. This confirms the benchmark covers a wide range of cognitive difficulty levels."

    Reasoning depth D(T) is defined in Eq. (1) as the count of required atomic reasoning/interaction steps from checklist C, i.e., it is the paper's own operationalization of cognitive difficulty for a CAPTCHA. Observing a spread in annotated depth values is therefore the very definition of a spread in the measured difficulty quantity, so describing the value 2.94 ± 0.92 as 'confirms ... a wide range of cognitive difficulty levels' is a tautological restatement of the annotation distribution rather than an independent confirmation of difficulty diversity. This step is not load-bearing for the central empirical claim: the headline 40.0% vs 93.3% gap in Table 1 comes from independent browser-loop runs and is not computed from or fitted to the depth values, so no prediction reduces to the metric.

full rationale

The paper's central claim is an empirical contrast: browsers-use agents at most 40.0% (Openai-o3) versus humans at 93.3% (Table 1), both measured directly in the testing platform rather than derived from any equation. Nothing is fitted to a subset and then 'predicted': the success rates are run outcomes, the cost figures are billing records, and the CAPTCHA Reasoning Depth metric is used descriptively (distribution, behavioral comparison) without any regression or formula that maps depth to success. The only overlapping-author citation is MathVista [21] (a co-named Jiacheng Liu), used in the Related Work list as an example of a math benchmark; it is not invoked to justify any CAPTCHA claim, so it is not load-bearing. The one tautological phrase (Section 3.2, 'This confirms the benchmark covers a wide range of cognitive difficulty levels') is self-verifying because depth is defined as the difficulty measure, but it does not feed into the headline result, keeping the overall score at 1. Two evidentiary gaps are correctness risks rather than circular reductions: (i) the 93.3% human figure is quoted without any protocol (number of participants, puzzles per participant, naivety, or attempt limits), and Section 3.3 uses that same figure to vouch for ground-truth reliability ('as humans can perform a 93.3% success rate'), which would be partially self-corroborating if the measured humans were the annotators who designed and vetted the puzzles - the paper never states that they were, so the reduction cannot be exhibited; (ii) per-type single-pass presentation makes point estimates like 40.0% = 8/20 trials and non-integer values like GPT-4o's 5.7% unverifiable without run counts or confidence intervals. The 'models overthink' analysis additionally rests on real verbatim traces (Fig. 4), giving it independent textual support. Overall the benchmark results stand as independent measurements; the reasoning-depth metric is an annotation tool with no predictive loop.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or physical entities. Its main intellectual commitments are the reasoning-depth annotation protocol and the assumption that the synthetic CAPTCHA set is a valid testbed for real-world agent readiness. These are domain assumptions rather than fitted quantities.

assumptions (4)
  • domain assumption Synthetic CAPTCHA puzzles are representative of real-world CAPTCHAs encountered by web agents.
    The entire benchmark and its conclusions rest on this; the paper does not validate against real CAPTCHA services (Section 3.3).
  • domain assumption The CAPTCHA Reasoning Depth checklist and annotation protocol yield a consistent, meaningful complexity measure.
    The metric is defined by human annotation using a heuristic checklist, but inter-annotator agreement is mentioned but not reported numerically (Section 3.2).
  • ad hoc to paper Human annotator success rate of 93.3% is a reliable baseline.
    The paper does not report the number of participants, their selection criteria, or the exact testing conditions for the human baseline (Table 1).
  • domain assumption Testing each model once per CAPTCHA type (about 20 puzzles total) is enough to estimate the model's success rate on the benchmark.
    The evaluation appears to use a single episode per model through one instance of each type (Section 4.2), which yields pass@1 estimates without variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents." pith.science (2026). https://pith.science/paper/JLIPHWG3

@misc{pith2026250524878,
  author       = {Pith},
  title        = {Pith review of: Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLIPHWG3}},
  note         = {Machine review of arXiv:2505.24878}
}
read the original abstract

CAPTCHAs have been a critical bottleneck for deploying web agents in real-world applications, often blocking them from completing end-to-end automation tasks. While modern multimodal LLM agents have demonstrated impressive performance in static perception tasks, their ability to handle interactive, multi-step reasoning challenges like CAPTCHAs is largely untested. To address this gap, we introduce Open CaptchaWorld, the first web-based benchmark and platform specifically designed to evaluate the visual reasoning and interaction capabilities of MLLM-powered agents through diverse and dynamic CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225 CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth, which quantifies the number of cognitive and motor steps required to solve each puzzle. Experimental results show that humans consistently achieve near-perfect scores, state-of-the-art MLLM agents struggle significantly, with success rates at most 40.0% by Browser-Use Openai-o3, far below human-level performance, 93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing the limits of current multimodal agents and guiding the development of more robust multimodal reasoning systems. Code and Data are available at this https URL.

Figures

Figures reproduced from arXiv: 2505.24878 by the authors.

Figure 1
Figure 1. Open CaptchaWorld data distribution and MLLMs performance plot. Despite the explosion of agent benchmarks, most systematically filter out CAPTCHAs. Visu￾alWebArena [15] and AgentBench [19] simulate realistic environments but discard pages with CAPTCHAs [43]. Traditional CAPTCHA-solving work (e.g., Deep-CAPTCHA [26], Breaking reCAPTCHAv2 [30]) treats them as static perception tasks solvable by CNNs or object detector… view at source ↗
Figure 2
Figure 2. Examples from Open CaptchaWorld. evaluate a wide range of MLLM capabilities. However, most assume a static, single-turn setup [45], limiting their ability to test dynamic, real-world interaction. To overcome this, recent work has explored LLM and MLLM agents operating in interactive en￾vironments [29, 37, 33], often with external tool use [49, 5, 8, 17, 32] and multi-step decision￾making [44, 39, 34]. Benchmarks lik… view at source ↗
Figure 3
Figure 3. CAPTCHA Reasoning Depth Estimation by Human Annotators and Most Advanced [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Thinking Process Comparison When Estimating CAPTCHA Reasoning Depth between [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Open CaptchaWorld Date Curation Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Step-by-step reasoning process of Openai-o3 in successfully solving Image Matching. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Cost-performance trade-off among browser￾use agents. Each point represents a model, plotted by its evaluation cost (in log scale) and pass@1 success rate on Open CaptchaWorld. Openai-o3 achieves the highest success rate but incurs substantial cost, while models like Ge…
Figure 8
Figure 8. Figure 8: Representative Failure of Openai-o3 Across Challenging CAPTCHA Types. (a) Failure case with correct strategy but limited visual perception. (b) Failure case due to complex operational execution. (c) Failure case caused by misguided solution strategy based on irrelevant…
Figure 9
Figure 9. Figure 9: Notice that all the images for each CAPTCHA are not repeated. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Prompt for estimating CAPTCHA Reasoning Depth. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Prompt to Browser Use Agents for testing on Open CaptchaWorld. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Commercial solvers defeat puzzle captchas at near-perfect rates for pennies per thousand, while reCAPTCHA v3's resistance comes from browser-environment authenticity, not human-like behavior.

Reference graph

Works this paper leans on

54 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [15]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649, 2024

    arXiv 2024

  2. [1]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, James Tilsted, Karen Simonyan, João Carreira, Erich Elsen, Matthias Minderer, et al. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022

    arXiv 2022

  3. [2]

    Claude 3.7 Sonnet System Card

    Anthropic. Claude 3.7 Sonnet System Card. https://www.anthropic.com/ claude-3-7-sonnet-system-card, 2025. Technical report, February 2025

    work page 2025

  4. [3]

    Qwen-vl: A versatile vision–language model for perception, localization, and generation.arXiv preprint arXiv:2308.12966, 2023

    Chunyuan Chen, Hao Li, Zhengxuan Liu, Yong Wang, Yi Zhou, Hangbo Li, Yue Li, Zhirui Liu, and Furu Wei. Qwen-vl: A versatile vision–language model for perception, localization, and generation.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  5. [4]

    Internvl: Scaling up vision foundation models and aligning for generic vision–language understanding.arXiv preprint arXiv:2312.14238, 2023

    Zihao Dou, Feng Wang, Lin Zhang, Shidong Liu, Shuai Lu, Luming Ding, Wengang Wang, Bo Wang, Lei Li, and Song Bai. Internvl: Scaling up vision foundation models and aligning for generic vision–language understanding.arXiv preprint arXiv:2312.14238, 2023

    arXiv 2023

  6. [5]

    Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    arXiv 2025

  7. [6]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  8. [7]

    Gemini 2.5 pro: Our most intelligent ai model

    Google DeepMind. Gemini 2.5 pro: Our most intelligent ai model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ ,

    work page 2025

Show all 54 references
  1. [8]

    Tora: A tool-integrated reasoning agent for mathematical problem solving

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 2023

    2023 arXiv

  2. [9]

    Making the v in VQA matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Daniel Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in VQA matter: Elevating the role of image understanding in visual question answering. InCVPR, 2017

    2017

  3. [10]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 10

    2017

  4. [11]

    Pc agent: While you sleep, ai works–a cognitive journey into digital world.arXiv preprint arXiv:2412.17589, 2024

    Yanheng He, Jiahe Jin, Shijie Xia, Jiadi Su, Runze Fan, Haoyang Zou, Xiangkun Hu, and Pengfei Liu. Pc agent: While you sleep, ai works–a cognitive journey into digital world.arXiv preprint arXiv:2412.17589, 2024

    2024 arXiv

  5. [12]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering.arXiv preprint arXiv:1902.09506, 2019

    1902 arXiv

  6. [13]

    FUNSD: A dataset for form understanding in noisy scanned documents.arXiv preprint arXiv:1905.13538, 2019

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. FUNSD: A dataset for form understanding in noisy scanned documents.arXiv preprint arXiv:1905.13538, 2019

    1905 arXiv

  7. [14]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2024

    2024 arXiv

  8. [16]

    Hoi, Steven C.˙Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Caiming Xiong, and H. Hoi, Steven C.˙Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  9. [17]

    Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    2025 arXiv

  10. [18]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Simon Jenni, and Jia Deng. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  11. [19]

    Agentbench: Evaluating LLMs as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, and et al. Agentbench: Evaluating LLMs as agents.arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  12. [20]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  13. [21]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    2024

  14. [22]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Alexander Schwing. Ok-vqa: A visual question answering benchmark requiring external knowledge. InCVPR, 2019

    2019

  15. [23]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. DocVQA: A dataset for visual question answering on document images. InWACV, 2021

    2021

  16. [24]

    Browser use: Enable ai to control your browser, 2024

    Magnus Müller and Gregor Žuniˇc. Browser use: Enable ai to control your browser, 2024. URL https://github.com/browser-use/browser-use

    2024

  17. [25]

    Webgpt: Browser- assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2022

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, et al. Webgpt: Browser- assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2022

    2022 arXiv

  18. [26]

    Deep-captcha: a deep learning based captcha solver for vulnerability assessment, 2020

    Zahra Noury and Mahdi Rezaei. Deep-captcha: a deep learning based captcha solver for vulnerability assessment, 2020

    2020

  19. [27]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf ,

  20. [28]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, , et al

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, , et al. Gpt-4o system card, 2024

    2024

  21. [29]

    Technical report, April 2025

    2025

  22. [30]

    Breaking recaptchav2

    Andreas Plesner, Tobias V ontobel, and Roger Wattenhofer. Breaking recaptchav2. In2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), page 1047–1056. IEEE, July 2024. doi: 10.1109/compsac61105.2024.00142. URL http://dx.doi.org/10. 1109/COMPSAC6110...

    2024 arXiv

  23. [31]

    Autoplan: Automatic planning of interactive decision-making tasks with large language models

    Siqi Ouyang and Lei Li. Autoplan: Automatic planning of interactive decision-making tasks with large language models. InFindings of EMNLP, 2023. 11

    2023

  24. [32]

    Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    2025 arXiv

  25. [33]

    Plummer, Liwei Wang, Cristina Cervantes, Juan C

    Bryan A. Plummer, Liwei Wang, Cristina Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30K entities: Collecting region-to-phrase correspondences for richer image–to–sentence models. InICCV, 2015

    2015

  26. [34]

    Autogpt: An autonomous gpt-4 experiment

    Significant Gravitas. Autogpt: An autonomous gpt-4 experiment. https://github.com/ Significant-Gravitas/AutoGPT, 2023. GitHub repository

    2023

  27. [35]

    Stateact: Enhancing llm base agents via self-prompting and state-tracking.arXiv preprint arXiv:2410.02810, 2024

    Nikolai Rozanov and Marek Rei. Stateact: Enhancing llm base agents via self-prompting and state-tracking.arXiv preprint arXiv:2410.02810, 2024

    2024 arXiv

  28. [36]

    Partially observable markov decision processes

    Matthijs TJ Spaan. Partially observable markov decision processes. InReinforcement learning: State-of-the-art, pages 387–414. Springer, 2012

    2012

  29. [37]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019

    2019

  30. [38]

    Phrasecut: Language grounding in images by text-based mask segmentation.ECCV, 2020

    Chenyun Wang, Xiaohui Shen, Zhicheng Lin, and Scott Cohen. Phrasecut: Language grounding in images by text-based mask segmentation.ECCV, 2020

    2020

  31. [39]

    Adaplanner: Adaptive planning from feedback with language models.arXiv preprint arXiv:2305.16653, 2023

    Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models.arXiv preprint arXiv:2305.16653, 2023

    2023 arXiv

  32. [40]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024

    2024 arXiv

  33. [41]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  34. [42]

    An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025

    2025

  35. [43]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Liang Wei, Jiaxing Zhang, Yue Wang, Meiyu Liu, Zhi Hu, Yiming Wang, Shikun Wang, Ziqi Zhang, Xingtian Dong, and Long Zhou. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  36. [44]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  37. [45]

    An illusion of progress? assessing the current state of web agents, 2025

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents, 2025

    2025

  38. [46]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling relationships in referring expressions with compositional modular networks. InCVPR, 2016

    2016

  39. [47]

    Survey on evaluation of llm-based agents, 2025

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. Survey on evaluation of llm-based agents, 2025

    2025

  40. [48]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  41. [49]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 12

    2023 arXiv

  42. [50]

    PubLayNet: Largest dataset ever for document layout analysis.arXiv preprint arXiv:1908.07836, 2019

    Xu Zhong, Jianbin Tang, and Antonio Jimeno-Yepes. PubLayNet: Largest dataset ever for document layout analysis.arXiv preprint arXiv:1908.07836, 2019

    1908 arXiv

  43. [51]

    Nemotron-research-tool-n1: Tool-using language models with reinforced reasoning.arXiv preprint arXiv:2505.00024, 2025

    Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemotron-research-tool-n1: Tool-using language models with reinforced reasoning.arXiv preprint arXiv:2505.00024, 2025

    2025 arXiv

  44. [52]

    ✓” indicates that the model demonstrated at least partial success on that type, while “✗

    Damo Zhu, Junyang Chen, Junnan Yang, Weijie Xu, Heyang Zhang, Jianxin Zhang, Yan Zhang, and Jianlong Chen. Minigpt-4: Enhancing vision–language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 13 Appendix A More Examples from Open Captch...

    2023 arXiv

  45. [53]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2024

    2024 arXiv

  46. [2025]

    Blog post, March 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.