Pith. sign in

REVIEW 4 major objections 6 minor 54 references

FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FormFactory benchmark shows six state-of-the-art MLLMs all fail realistic form filling, with no model above 5% accuracy and end-to-end completion below 2%.

desk verdict A genuinely useful benchmark resource undercut by an evaluation protocol that measures pixel-clicking rather than form-filling; the headline failure rate doesn't support the paper's conclusion. read the letter →

arxiv 2506.01520 v1 pith:AI63BE4S submitted 2025-06-02 cs.CL

Bobo Li , Yuheng Wang , Hao Fei , Juncheng Li , Wei Ji , Mong-Li Lee , Wynne Hsu This is my paper
classification cs.CL
keywords form-fillingagentsmultimodallargelanguagemodelsGUIbenchmarkvisuallayoutreasoningfield-valuealignmentzero-shotevaluationwebautomation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that filling real-world web forms is a distinct capability that current multimodal large language models do not have, and that the gap deserves its own measurement tool. It introduces FormFactory, an interactive benchmark suite with 25 realistic forms, 1,250 input instances paired with 13,800 field-value annotations, and a backend scorer that checks submitted values against ground truth. In zero-shot evaluations of six state-of-the-art MLLMs, the paper reports that no model exceeds 5% accuracy, with click success rates below 10% and end-to-end form completion below 2%. The authors present these numbers as evidence that the models fail at visual layout reasoning and field-value alignment, not just at following instructions.

What carries the argument

The machinery that carries the argument is FormFactory itself: a Python-based interactive web platform that serves forms, a dataset generated from LLM-synthesized and real documents with ground-truth key-value pairs, and a backend scorer that computes atomic and episodic metrics. The task is formalized as a page-level sequential decision process (c, E, S, A) in which the agent reads the user input and screenshot state and outputs Click(x,y) and Type(text) actions, executed by an automation tool. A Ruler-Enhanced Strategy overlays pixel-scale axes along the horizontal and vertical edges of the screenshot to give the model geometric references for coordinate prediction.

What would settle it

Run the same 25 FormFactory forms with a human baseline and with a non-coordinate agent that selects fields via the DOM or accessibility tree; if either completes the forms at high accuracy while the pixel-coordinate agents stay near zero, the low scores trace to the coordinate interface rather than to a missing form-filling capability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that form filling is a much harder problem for current MLLMs than generic GUI use. Given a page screenshot, a user input document, and the freedom to emit Click(x,y) and Type(text) actions, GPT-4o, Gemini 2.5 Pro, Claude 3.7 Sonnet, Qwen-VL-Max, Grok 3, and Doubao-Vision-Pro-32k all score below 5% on the benchmark's overall accuracy in zero-shot settings. Click accuracy is almost always below 10%, and no model completes end-to-end forms more than 2% of the time. Value accuracy is much higher—often above 70–90%—but the paper explains that its value metric counts a field as correct if the value appears somewhere in the output, independent of whether it was typed into the right place.

Load-bearing premise

The result rests on treating Click(x,y) with absolute pixel coordinates as a fair way to measure form-filling ability; since MLLMs are not trained to predict pixel-level positions with precision, the low click accuracy may be an interface mismatch rather than a deficit in form-filling reasoning.

Editorial extensions

If this is right

  • An agent that solves FormFactory would need to combine semantic value extraction with precise spatial grounding, because value accuracy is already high while click accuracy is low.
  • Because all six evaluated models fail in the same pattern, progress will likely come from better visual grounding or a more model-appropriate interaction space rather than from stronger language reasoning alone.
  • The ruler ablation shows that lightweight visual references help on simple forms but not complex ones, so layout density and field diversity are real scalability bottlenecks.
  • FormFactory gives future work a standardized zero-shot protocol for comparing form-filling agents and for tracking whether new models improve end-to-end completion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A human baseline on the same 25 forms would calibrate the benchmark; if humans with normal mouse control score near 100%, then the interface is not inherently unfair, and if they also struggle, the task may be underspecified.
  • Swapping the action space from absolute pixel coordinates to DOM-node or accessibility-tree references while keeping the same forms and scorer would separate reasoning from coordinate prediction; a large jump in click accuracy would suggest the paper's conclusion overstates the reasoning deficit.
  • Because the dataset is synthetic and released, it can be used for supervised learning of form-filling trajectories, and a fine-tuned vision-language model that exceeds the 5% bar would show whether the current ceiling is a training or an interface problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. FormFactory introduces a benchmark for multimodal form-filling agents: a Flask-based interactive web platform, 25 forms across 8 domains, and a dataset of 1,250 instances with 13,800 field-value annotations. The paper evaluates several commercial MLLMs in a zero-shot setting by having each model output sequences of Click(x,y) and Type(text) actions, which are executed on a Windows desktop via PyAutoGUI. The reported results show low click accuracy (mostly below 10%) and higher value scores, and the authors conclude that current MLLMs lack the visual layout reasoning and field-value alignment needed for form filling, claiming 'no model surpasses 5% accuracy' and 'end-to-end form completion rates remain under 2%'.

Significance. A dedicated, reproducible benchmark for form-filling agents is a useful contribution, and the interactive platform with backend evaluation could serve the community if the evaluation protocol is sound. However, the paper's central finding is a negative capability claim about state-of-the-art MLLMs, and that claim rests on metrics and an action space whose validity is not established. The authors are transparent about two key limitations—the Value metric does not verify that values landed in the correct fields (Section 5.2), and MLLMs are not trained to predict pixel-level coordinates (Section 4.2)—yet the abstract and conclusion still draw broad conclusions about layout reasoning and field-value alignment. If the protocol were calibrated (e.g., with a human baseline or a DOM-grounding condition), the benchmark would be substantially stronger; as presented, its headline results are not supported by the reported evidence.

major comments (4)
  1. [Section 5.2, Table 2] The Value metric is explicitly defined not to verify field-value alignment: 'we consider a field correct as long as its corresponding value appears in the model’s output, regardless of whether it was actually entered into the correct UI location.' Consequently, the high Value scores (e.g., 70–99% on String and Drop-down for several models) measure whether the required text is generated anywhere in the model's output, not whether the form was filled correctly. The conclusion that models lack 'field-value alignment abilities' is therefore unsupported by this metric; in fact, the metric design excludes alignment by construction. Any episodic score derived from Value inherits this problem. This is load-bearing for the central claim about form-filling failure.
  2. [Sections 3.3, 4.1, 4.2] The action space requires models to emit absolute pixel coordinates Click(x,y), which are then executed on the live desktop via PyAutoGUI. The paper itself states in Section 4.2 that modern MLLMs 'are not trained to predict pixel-level positions with precision,' and the Ruler-Enhanced Strategy is introduced precisely to mitigate this. Low click success rates under this protocol conflate a known input-format deficiency (coordinate regression) with a deficit in visual layout reasoning or form-filling competence. No human baseline, no alternative action space (e.g., DOM element IDs or accessibility-tree grounding), and no comparison against existing GUI benchmarks (Mind2Web, WebArena, OSWorld) are provided. Without at least one of these, the claim that models lack visual layout reasoning is not established; the data only show that models cannot emit sufficiently precise absolute coordinates in this specific setup.
  3. [Abstract, Section 6, Table 2] The headline numbers in the abstract and conclusion—'no model surpasses 5% accuracy' and 'end-to-end form completion rates remain under 2%'—do not follow from any reported metric. Table 2 reports separate atomic and episodic Click and Value values; episodic Value scores for several models are 58–73%, and the episodic Click values are not completion rates. The paper never defines what counts as an 'episodic success' (e.g., whether all backend-stored values must match the ground truth, whether the model must click every field, or whether values may appear anywhere in the output). Without a precise definition and the corresponding number in the results, the paper's two flagship quantitative claims are unverifiable.
  4. [Section 3.2, Table 1] The dataset is described as 'realistic' and 'high-fidelity,' but most instances are LLM-synthesized; only the academic-submission forms use real papers as source material. There is no human validation of the generated key-value annotations, no inter-annotator agreement, and no estimate of human success rate on the same tasks. Without a human baseline or an estimate of feasible performance, the claim that a 'striking mismatch' exists between MLLM capabilities and task demands is not calibrated. A human baseline on the same pixel-click protocol would also help separate intrinsic difficulty from the interface artifact identified above.
minor comments (6)
  1. [Sections 1, 3.1, Table 1] The paper inconsistently states that the benchmark contains '25 diverse forms' (Section 1) and '20 realistic forms' (Section 3.1); Table 1 lists 25 forms. Please harmonize the counts.
  2. [Table 2] The Description Value for GPT-4o is 0.8407 (a BLEU score in [0,1]) while all other Value entries are percentages in [0,100]. Add a footnote explaining the scale difference, and consider converting BLEU to a percentage or reporting it separately.
  3. [Figure 4] The caption states 'Group 0 reflects overall performance,' which is confusing because groups 1–5 are condition labels. Specify how groups are defined and whether error bars represent variance across the 25 forms (or across instances), and report the number of samples per group.
  4. [Section 4.2] The Ruler-Enhanced Strategy overlays pixel-scale axes on the screenshot, but it is not clear whether this overlay is part of the benchmark interface or a per-model prompting modification. This affects reproducibility and whether the ruler is a benchmark feature or an evaluation intervention; please clarify.
  5. [Sections 5.4.1, 5.4.2] Click accuracy is reported only for String and Description fields because performance on other field types is 'near zero.' This selection is reasonable, but it should be stated in the main results (not only in the analysis section), and the claim that near-zero performance on dropdowns/checkboxes shows difficulty should be cross-referenced with the coordinate-output confound.
  6. [References] Several references are informal preprints or system cards without archive numbers or version dates; adding arXiv IDs or DOIs where available would improve reproducibility.

Circularity Check

1 steps flagged · score 5.0 of 10

Low click and completion rates are predetermined by the benchmark's absolute-pixel-coordinate action space once the paper admits MLLMs are not trained for pixel-level position prediction; the form-filling conclusion restates the task definition.

  1. self definitional [Section 3.3 (Task Definition), Section 4.2 (Ruler-Enhanced Strategy), Section 6 (Conclusion)]
    "The action space primarily includes: Click(x, y)— mouse click at pixel coordinates; Type(text). ... While modern Multimodal LLMs exhibit impressive reasoning capabilities, they are not trained to predict pixel-level positions with precision. ... click success rates fall below 10%, and end-to-end form completion rates remain under 2%. These results highlight that form-filling is a substantially harder task than standard GUI interaction, demanding fine-grained grounding, alignment, and reasoning."

    FormFactory defines the form-filling task as a sequence of absolute-pixel Click(x,y)/Type actions executed via PyAutoGUI, so success is measured by correct coordinate prediction. The paper then concedes in Section 4.2 that MLLMs are not trained for pixel-level position prediction. The reported sub-10% click rates and sub-2% completion rates therefore re-express a known limitation of the protocol, not an independently measured deficit in visual layout reasoning and field-value alignment. No human baseline or non-coordinate DOM/accessibility grounding is provided to separate interface mismatch from form-filling competence.

full rationale

The paper is a benchmark paper with no fitted parameters, no self-citation chains, and no imported uniqueness theorems; its dataset construction and evaluation pipeline are internally consistent. The circularity is confined to the interpretation of the headline failure rates. FormFactory defines the target task itself through absolute-pixel Click(x,y) actions (Section 3.3), and Section 4.2 explicitly states that MLLMs are not trained to predict pixel-level positions. Therefore, the conclusion that current models lack visual layout reasoning and field-value alignment, drawn from sub-5% accuracy and sub-2% completion rates, is at least partly a restatement of the benchmark's chosen action space rather than an independent external fact. The paper's own Section 5.2 caveat that Value scores do not require correct field placement reinforces this. The benchmark remains potentially useful, but the central capability claim is not calibrated by a human baseline or an alternative grounding channel, so the finding does not support the broad conclusion as stated. This is a moderate partial circularity, not a fabricated fit.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central finding of model failure rests on the coordinate-based action interface, the lenient Value metric, and an unvalidated synthetic dataset. These are the main axioms; the paper provides no external calibration against humans, other benchmarks, or non-coordinate agents.

free parameters (2)
  • Click correctness tolerance (pixel distance threshold) = unstated
    The benchmark declares a click correct based on an undocumented pixel-distance threshold; this threshold determines the reported near-zero click accuracy and is a free evaluation parameter.
  • Episodic success criterion = unstated
    The paper does not define whether episodic accuracy requires all fields clicked correctly, all values present, or partial credit; the reported episodic value (70.7%) and the claim of under 2% completion cannot be reconciled without this definition.
assumptions (3)
  • ad hoc to paper Evaluation by absolute pixel coordinate Click(x,y) executed via PyAutoGUI is a valid interface for form-filling agents.
    Introduced in Sections 3.3 and 4.1; no justification that this interface is representative of real form filling, and the authors admit in Section 4.2 that MLLMs are not trained for pixel-level position prediction.
  • domain assumption LLM-generated synthetic forms and descriptions provide accurate ground-truth annotations without human verification.
    Section 3.2 describes generating gold field values via LLM prompts; no inter-annotator agreement, human audit, or error analysis is reported.
  • ad hoc to paper The Value metric (presence of the value in model output) is a valid proxy for form-filling performance.
    Section 5.2 explicitly states correctness is judged by whether the value appears in output regardless of UI placement; the paper acknowledges a stricter protocol would lower scores.
invented entities (1)
  • GUI Ruler overlay
    purpose: An image augmentation adding pixel-scale axes to screenshots to help models predict click coordinates.
    Introduced in Section 4.2 and evaluated only via the paper's own ablation (Figure 4); no external validation that it measures or improves spatial reasoning beyond this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents." pith.science (2026). https://pith.science/paper/AI63BE4S

@misc{pith2026250601520,
  author       = {Pith},
  title        = {Pith review of: FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AI63BE4S}},
  note         = {Machine review of arXiv:2506.01520}
}
read the original abstract

Online form filling is a common yet labor-intensive task involving extensive keyboard and mouse interactions. Despite the long-standing vision of automating this process with "one click", existing tools remain largely rule-based and lack generalizable, generative capabilities. Recent advances in Multimodal Large Language Models (MLLMs) have enabled promising agents for GUI-related tasks in general-purpose scenarios. However, they struggle with the unique challenges of form filling, such as flexible layouts and the difficulty of aligning textual instructions with on-screen fields. To bridge this gap, we formally define the form-filling task and propose FormFactory, an interactive benchmarking suite comprising a web-based interface, backend evaluation module, and carefully constructed dataset. Our benchmark covers diverse real-world scenarios, incorporates various field formats, and simulates high-fidelity form interactions. We conduct a comprehensive evaluation of state-of-the-art MLLMs and observe that no model surpasses 5% accuracy, underscoring the inherent difficulty of the task. These findings also reveal significant limitations in current models' visual layout reasoning and field-value alignment abilities. We hope our benchmark can serve as a stepping stone for further research into robust, practical form-filling agents.

Figures

Figures reproduced from arXiv: 2506.01520 by the authors.

Figure 1
Figure 1. Overview of the form-filling task and its challenges. Compared to general GUI tasks, form-filling involves more [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example interface for a form filling task, using a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the form-filling task system. (a) The platform takes a form and a resume as input, prompting an MLLM [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Click accuracy with and without Ruler guidance [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Value accuracy across varying field counts (left, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 38 canonical work pages

  1. [1]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Bar- reira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binko...

    work page 2022

  2. [2]

    Anthropic. 2025. Claude 3.7 Sonnet System Card. (2025)

    work page 2025

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A frontier large vision- language model with versatile abilities.arXiv:2308.12966(2023)

    arXiv 2023

  4. [4]

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. 2024. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale.arXiv:2409.08264(2024)

    arXiv 2024

  5. [5]

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Shuai Ren, and Hongsheng Li. 2025. AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents.arXiv:2407.17490(2025)

    arXiv 2025

  6. [6]

    Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. 2025. A3: Android Agent Arena for Mobile GUI Agents.arXiv:2501.01149(2025)

    arXiv 2025

  7. [7]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, Yiqiang Li, Chenlong Wang, Yue Yu, Tianshuo Zhou, Zhen Li, Yi Gui, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. 2025. GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding. InProceedings of ICLR

    work page 2025

  8. [8]

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, and Kun Shao. 2025. Spa-Bench: a comprehensive Benchmark for Smartphone Agent Evaluation. InProceedings of ICLR

    work page 2025

Show all 54 references
  1. [9]

    Ke Chen, Zhe Zhang, Wen Zeng, Richang Zhang, Feng Zhu, and Rui Zhao

  2. [10]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. InProceedings of ACL. 9313–9332

    2024

  3. [11]

    Google DeepMind. 2025. Gemini 2.5: Our Most Intelligent AI Model. (2025)

    2025

  4. [12]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. InProceedings of NeurIPS

    2023

  5. [13]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. CogAgent: A Visual Language Model for GUI Agents. InProceedings of CVPR. 14281–14290

    2024

  6. [14]

    Weihan Hong, Wen Wang, Jui Tseng, Mike Lewis, Xisen Shi, and Xu Chang

  7. [15]

    Lillicrap

    Peter Conway Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam San- toro, and Timothy P. Lillicrap. 2022. A data-driven approach for learning to control computers. InProceedings of ICML. 9466–9482

    2022

  8. [16]

    LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention.arXiv:2303.16199(2023)

    2023 arXiv

  9. [17]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried

  10. [18]

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2024. OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web. InProceedings of ECCV. 161–178

    2024

  11. [19]

    Jia-Nan Li, Jian Guan, Wei Wu, Zhengtao Yu, and Rui Yan. 2025. 2D-TPE: Two- Dimensional Positional Encoding Enhances Table Understanding for Large Lan- guage Models. InProceedings of the ACM on Web Conference. 2450–2463

    2025

  12. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. InProceedings of ICML, Vol. 202. 19730–19742

    2023

  13. [21]

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. 2024. AutoWebGLM: A Large Language Model-based Web Navigating Agent. InPro- ceedings of KDD. 5295–5306

    2024

  14. [22]

    Kunyu Li, Yi He, Yinan Wang, Wei Li, Wen Wang, Ping Luo, Yu Wang, and Yang Qiao. 2023. VideoChat: Chat-centric video understanding.arXiv:2305.06355 (2023)

    2023 arXiv

  15. [23]

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen Wu, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. 2024. VideoGUI: A Benchmark for GUI Automation from Instructional Videos. InProceedings of NeurIPS

    2024

  16. [24]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: Bootstrap- ping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. InProceedings of ICML. 12888–12900

    2022

  17. [25]

    OpenAI. 2023. GPT-4 Technical Report.arXiv:2303.08774(2023)

    2023 arXiv

  18. [26]

    OpenAI. 2024. GPT-4o System Card.arXiv:2410.21276(2024)

    2024 arXiv

  19. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. InProceedings of NeurIPS

    2023

  20. [28]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. InProceedings of ACL. 311–318

    2002

  21. [30]

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. 2024. We- bCanvas: Benchmarking Web Agents in Online Environments.arXiv:2406.12373 (2024)

    2024 arXiv

  22. [31]

    Lillicrap

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P. Lillicrap. 2023. AndroidInTheWild: A Large-Scale Dataset For Android Device Control. InProceedings of NeurIPS

    2023

  23. [32]

    ByteDance Seed. 2024. Doubao-vision-pro-32K. (2024)

    2024

  24. [33]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InProceedings ...

    2021

  25. [34]

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang

  26. [35]

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI. In Proceedings of EMNLP. 6699–6712

    2022

  27. [36]

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. 2023. From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces. InProceedings of NeurIPS

    2023

  28. [37]

    Qwen Team. 2024. Introducing Qwen-VL. (2024)

    2024

  29. [38]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing Vision-Language Mode...

    2024 arXiv

  30. [39]

    xAITeam. 2024. Grok-3. (2024)

    2024

  31. [40]

    Al Sweigart. 2023. PyAutoGUI: Cross-platform GUI automation for human beings. https://pyautogui.readthedocs.io/en/latest/

    2023

  32. [41]

    Zhengyuan Yang, Lei Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Farhad Ahmed, Zicheng Liu, Cong Liu, Michael Zeng, and Lijuan Wang. 2023. MM-REACT: Prompting ChatGPT for multimodal reasoning and action. arXiv:2303.11381(2023)

    2023 arXiv

  33. [42]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. InProceedings of NeurIPS

    2022

  34. [43]

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. Large Language Model-Brained GUI Agents: A Survey.CoRR abs/2411.18279 (2024)

    2024 arXiv

  35. [44]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu

  36. [45]

    InProceedings of NeurIPS

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. InProceedings of NeurIPS

  37. [46]

    Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, and Jianwei Yin. 2024. GUI Testing Arena: A Unified Bench- mark for Advancing Autonomous GUI Testing Agent.arXiv:2412.18426(2024)

    2024 arXiv

  38. [47]

    Weichao Zhao, Hao Feng, Qi Liu, Jingqun Tang, Binghong Wu, Lei Liao, Shu Wei, Yongjie Ye, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. 2024. TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy. InProceedings of NeurIPS

    2024

  39. [48]

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. 2024. Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control. InPro- ceedings of ICLR

    2024

  40. [49]

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024. Android in the Zoo: Chain-of-Action-Thought for GUI Agents. InFindings of EMNLP. 12016–12031

    2024

  41. [50]

    Zhuosheng Zhang and Aston Zhang. 2024. You Only Look at Screens: Multimodal Chain-of-Action Agents. InFindings of ACL. 3132–3149

    2024

  42. [54]

    Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal Table Understanding. InProceedings of Conference’17, July 2017, Washington, DC, USA Li et al. ACL. 9102–9124

    2024

  43. [55]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. InProceedings of ICLR

    2024

  44. [2017]

    In Proceedings of ICML

    World of Bits: An Open-Domain Platform for Web-Based Agents. In Proceedings of ICML. 3135–3144

  45. [2023]

    arXiv:2306.15195(2023)

    Shikra: Unleashing multimodal LLM’s referential dialogue magic. arXiv:2306.15195(2023)

    2023 arXiv

  46. [2024]

    InProceedings of ACL

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. InProceedings of ACL. 881–905

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.