Pith. sign in

REVIEW 4 major objections 5 minor 52 references

IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces the first public benchmark for using large language models to control real industrial software through its graphical user interface, and measures how far current models are from being usable.

desk verdict A genuinely useful new dataset for industrial GUI control, but the evaluation chain has unvalidated links that need to be fixed before the benchmark numbers can be trusted. read the letter →

arxiv 2509.01199 v1 pith:UAJMTF2Y submitted 2025-09-01 eess.SY cs.SY

Xiaoran Yang , Yuyang Du , Kexin Chen , Soung Chang Liew , Jiamin Lu , Ziyu Guo , Xiaoyan Liu , Qun Yang
show 9 more authors
Shiqi Xu Xingyu Fan Yuchen Pan Taoyong Cui Hongyu Deng Boris Dudder Jianzhang Pan Qun Fang Pheng Ann Heng
This is my paper
classification eess.SYcs.SY
keywords generalcomputercontrolindustrialautomationGUILLMagentsbenchmarkdatasetPyAutoSmith-Watermanalignmentmultimodalhumaninteractiondata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents IndusGCC, a dataset and evaluation framework claiming to be the first LLM-based General Computer Control benchmark built for industrial environments. It contains 448 real-world tasks across seven industrial domains, each paired with synchronized screen recordings, mouse positions, mouse and keyboard events, and human-written task descriptions. The authors propose four complementary metrics to score generated control scripts: functional equivalence judged by an LLM, sequence similarity via Smith-Waterman alignment, per-operation hit rate with spatial tolerances, and operation redundancy. Benchmarking eight mainstream text-based LLMs shows that even the best model succeeds on only about 22% of tasks in its strongest subset, with many subsets near zero. If the benchmark is sound, it gives the field a shared resource for measuring and improving AI agents that operate closed-source industrial GUIs.

What carries the argument

The benchmark's core apparatus is the comparison between a generated TestCode script and a gold-standard SampleCode script, both written in PyAutoGUI. The gold standard is produced by a multi-agent pipeline: three LLMs independently generate candidate scripts from recorded human interactions, the two most similar candidates are selected by Smith-Waterman alignment, and a fourth LLM merges them under human review. Evaluation uses four metrics: an LLM-as-judge functional-equivalence check for task success, Smith-Waterman alignment over AST-derived operation sequences for structural similarity, per-operation hit rate with click tolerance regions, and a redundancy rate measuring extra or missing

What would settle it

Take a random sample of, say, 50 IndusGCC tasks, have a human operator execute each gold-standard SampleCode script in the corresponding factory software, and check whether the intended final system state is reached. If a substantial fraction of the scripts fail to reproduce the recorded outcome, the reference code cannot support the reported task success rates.

Watch

Extended reading notes

Core claim

IndusGCC is positioned as the first dataset and benchmark tailored to LLM-based General Computer Control in industrial settings. The central empirical claim is that current mainstream LLMs cannot yet act as reliable plug-and-play controllers for industrial equipment via GUI automation. Across seven real-world domains—robotic arm control, network device configuration, software-defined radio, chemical synthesis, welding control, robot path planning, and network traffic analysis—the best task success rate is 22.73% (on the chemical subset), and several subsets fall to 0%. Sequence similarity scores are notably higher, indicating that models often reproduce the structure of human interaction seq

Load-bearing premise

The gold-standard scripts used as ground truth are generated by a multi-agent LLM pipeline and then human-reviewed, so if those scripts do not actually reproduce correct, executable human operations, all success and similarity scores measure agreement with an unreliable reference rather than real operational correctness.

Editorial extensions

If this is right

  • A public benchmark now exists for measuring LLM performance on closed-source industrial GUI control, filling a gap left by web- and mobile-focused GCC benchmarks.
  • The four metrics separate final task success from procedural fidelity, enabling future work to diagnose whether models fail at perception, planning, or execution.
  • The recorded multimodal human interaction data can serve as supervision for training or fine-tuning models on industrial GUI operation, not only for evaluation.
  • The explicit click-tolerance annotations prevent unfair penalization of practically acceptable actions, a design that may transfer to other precision-sensitive GUI benchmarks.
  • The low success rates across all eight tested models set a clear baseline, so future systems can demonstrate progress against a concrete, reproducible starting point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's observation setup—raw screen pixels plus task description—extends naturally to other closed-source enterprise software where accessibility APIs and DOM trees are unavailable, such as medical imaging or avionics maintenance.
  • The LLM-as-judge functional-equivalence metric could be validated independently by executing a sample of generated scripts in a mirrored or live environment and comparing observed system states with human judgment.
  • The gold-standard generation rule assumes that agreement among two LLMs indicates correctness; a systematic blind spot shared by all three generators would silently corrupt the reference, so adding human-executed verification on a subset would strengthen the benchmark.
  • A testable extension suggested by the redundancy findings: models that add extra clicks as a 'safety margin' may reduce task success when those clicks trigger unintended state changes, which could be measured by correlating redundancy rate with failure modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces IndusGCC, a dataset and benchmark for LLM-based General Computer Control (GCC) in industrial automation. It contains 448 real-world tasks across seven industrial domains, with synchronized screen recordings, mouse events, keyboard events, and task descriptions. The authors propose an evaluation framework with four metrics: Task Success Rate (via a GPT-4o judge), Sequence Similarity Score (Smith-Waterman on operation sequences), Operation Hit Rate, and Operation Redundancy Rate. They benchmark eight text-based LLMs using a preprocessing pipeline based on frame differencing, OCR, and PyAutoGUI code generation. The central claim is that IndusGCC is the first dataset/benchmark for GUI-based LLM control in industrial settings, and the experimental results show that current LLMs achieve low task success rates, indicating a significant gap for future research.

Significance. If the reference scripts and evaluation metrics are trustworthy, IndusGCC would be a valuable shared resource: it addresses a realistic and under-served setting (closed-source industrial GUIs with no structured access), provides real factory data, and benchmarks an accessible PyAutoGUI-based control formulation. The dataset is publicly released, which supports reproducibility. However, the current manuscript does not establish the validity of the gold-standard reference code or the LLM-as-judge evaluation, and the reported numbers lack uncertainty quantification. These gaps directly affect the benchmark's central value as a measure of operational correctness.

major comments (4)
  1. [Section 3.4, Ground Truth Code Generation] The gold-standard SampleCode is generated by an unvalidated LLM pipeline: three LLMs produce candidate scripts, the most similar pair is selected via Smith-Waterman using ad-hoc scores (Eqs. 1-3), and DeepSeek-V3 merges them. The manuscript states the outputs were 'reviewed by human experts' but does not specify the number of reviewers, their criteria, correction rate, or independence. Critically, the paper does not report that SampleCode was ever executed against the actual industrial GUI or validated against the recorded human interaction data. Since every metric in Section 4.1 compares TestCode to SampleCode, a hallucinated or semantically plausible but wrong reference operation propagates into all reported results. Please validate at least a stratified sample of reference scripts by executing them in the environment (or against the recorded human mouse/keyboard ground truth) and repo
  2. [Section 4.1, Task Success Rate] The GPT-4o judge is used to determine whether TestCode and SampleCode are functionally equivalent, but no validation of this judge is reported: no human agreement study, no confidence scores, no error analysis, and no comparison with an alternative judge. Because SampleCode is itself LLM-generated, the judge may systematically prefer outputs that resemble LLM-generated code. Moreover, Task Success Rate is defined as equivalence to SampleCode, not as achievement of the intended system state. Please provide judge validation on a subset (e.g., human agreement, disagreement examples, or state-based verification) and report the results.
  3. [Section 4.1 / Table 1] All reported metrics are point estimates without error bars, confidence intervals, or significance tests. With only 22-147 tasks per subset, observed differences such as claude-sonnet-4 vs. deepseek-v3 on AP may not be robust. Statements like 'claude-sonnet-4 clearly dominated' require uncertainty quantification. Please include bootstrap confidence intervals, per-task score distributions, or statistical significance tests to support the benchmarking conclusions.
  4. [Section 3.4, Eq. (3)] The definition of SimilarityScore is not mathematically clear. The denominator 'n * score(a_i^T, a_j^S) | a_i^T != a_j^S' is undefined: is n the length of AS? What does the conditional evaluate? The text then states 'The final Smith-Waterman score is normalized by the length of the reference sequence', which conflicts with the formula as written. Since Sequence Similarity Score is a headline metric, please provide a precise, self-contained definition and justify the substitution score and gap penalty, or include a sensitivity analysis for these parameters.
minor comments (5)
  1. [Section 2] Typo: 'data types that is allowed' should be 'data types that are allowed'.
  2. [Figure 2] Some statistics appear duplicated in the figure text (e.g., '#Sum – 466 #Avg – 6.96' appears twice for the first subset). Please correct the figure.
  3. [Section 4.1] Operation Redundancy Rate can be negative, but the definition 'proportion of extraneous operations relative to the number of valid steps' is ambiguous for negative values. Please specify the exact formula, including the denominator when TestCode has fewer operations than SampleCode.
  4. [Section 5] Grammar: 'even though introduce inefficiency' should be 'even though they introduce inefficiency'.
  5. [References] Reference [46] contains placeholder text 'Version: X.Y.Z, Accessed: INSERT_DATE_HERE'. Please complete the reference.

Circularity Check

2 steps flagged · score 4.0 of 10

Evaluation metrics reduce to agreement with an LLM-generated SampleCode; task success is self-referential rather than independently verified.

  1. self definitional [Section 4.1, 'Task Success Rate'; Section 3.4, 'Ground Truth Code Generation']
    "Task Success Rate: The first dimension measures whether TestCode and SampleCode are functionally equivalent in accomplishing the intended task. To evaluate this, we employ GPT-4o as an automatic judge: it receives both scripts together with the task description and determines whether they can be regarded as semantically equivalent."

    SampleCode is produced in Section 3.4 by a multi-agent LLM pipeline (GPT-4o, Gemini-2.5-Pro, Claude 4 Sonnet generate candidates; DeepSeek-V3 merges the most similar pair) and is never executed against the real industrial GUI. Task Success Rate is therefore, by construction, the degree to which TestCode matches an LLM-generated reference according to another LLM judge. The metric does not independently verify that either script produces the intended system state; it only measures agreement with the LLM consensus. The paper's claim that this metric 'reflects end-to-end task success' thus reduces to 'agrees with the LLM-generated gold standard'—a self-referential definition unless the SampleCode itself is externally validated.

  2. other [Section 3.4, Cooperative Code Generation]
    "This is motivated by our observation that if two LLMs reach agreement on the operation sequence (i.e., mouse clicking and keyboard input), then the resulting code is very likely to be the correct one."

    The gold-standard reference is selected by assuming that LLM agreement equals correctness. The same LLMs (GPT-4o, Claude 4 Sonnet, DeepSeek-V3) are later evaluated against this reference, so part of the evaluation measures models against output that they themselves helped generate. The 'agreement implies correctness' assumption is asserted without execution or independent verification, making the benchmark's target LLM consensus rather than objectively verified industrial operation.

full rationale

IndusGCC has independent value as a dataset: 448 real-world tasks, multimodal human interaction recordings, and manual task segmentation/review are not derived from the evaluation framework. However, the load-bearing evaluation chain is self-referential. Section 3.4 constructs SampleCode via three LLMs, selects the most similar pair with Smith-Waterman, and merges with DeepSeek-V3; the result is 'reviewed by human experts' but never executed. Section 4.1 then defines all four metrics—Task Success Rate, Sequence Similarity, Operation Hit Rate, and Redundancy—as comparisons against this SampleCode, with GPT-4o as judge for the primary metric. Thus the reported 'task success' equals agreement with an LLM-generated reference according to an LLM judge, not verified equipment control. The human review is mentioned but not specified (number of experts, criteria, correction rate), and the 'agreement implies correctness' heuristic is an unverified assumption. This does not invalidate the dataset or the framework entirely—the data collection and task segmentation are independent—but the central success metric is partially circular, measuring LLM consensus rather than an externally validated ground truth. The paper's self-citations to prior group work (e.g., Chemist-X) are used for tools, not to justify the evaluation's correctness, so they are not the main source of circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The load-bearing assumptions are procedural: that LLM-generated gold code is a trustworthy ground truth, that an LLM judge reliably scores functional equivalence, and that the video preprocessing pipeline is accurate enough for the benchmark to measure what it claims. The free parameters are evaluation hyperparameters and annotation tolerances that influence the reported scores.

free parameters (3)
  • Smith-Waterman substitution score and gap penalty = score match=2, mismatch=-1, gap delta=-1
    These scoring constants in Equation (2) are chosen by hand and affect the sequence similarity and redundancy metrics. They are not derived from data or calibrated against human judgments.
  • Frame differencing thresholds and clustering parameters (Kmeans+DBSCAN) = not reported in main text
    The visual observation phase depends on thresholds and clustering hyperparameters, partly inherited from the cited Chemist-X tool [6], which are not specified or tuned in this paper.
  • Tolerance regions for mouse operations = not reported
    The tolerance regions are annotated per task but the paper does not report the annotation criteria, inter-annotator agreement, or the distribution of tolerance sizes. These regions directly determine the Operation Hit Rate.
assumptions (3)
  • domain assumption LLM-generated reference code, after human review, is a valid gold standard for the human's observed operation.
    Invoked in Section 3.4: the gold-standard code is produced by three LLMs plus DeepSeek-V3 merging, then reviewed by human experts. The paper provides no quantitative validation (e.g., human agreement rate, execution verification) that the resulting code actually reproduces the recorded operation.
  • domain assumption GPT-4o as an LLM judge reliably determines functional equivalence between generated and reference scripts.
    Invoked in Section 4.1 for the Task Success Rate. No validation of the judge is reported, no confidence scores, no human agreement study, and no analysis of judge errors.
  • domain assumption The video preprocessing pipeline (frame differencing, Kmeans+DBSCAN pointer detection, EasyOCR) produces accurate keyframes and click coordinates from raw screen recordings.
    Invoked in Section 4.2 and Appendix A. The paper reports model errors are caused by this pipeline, but provides no ablation or accuracy measurement of the preprocessing itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation." pith.science (2026). https://pith.science/paper/UAJMTF2Y

@misc{pith2026250901199,
  author       = {Pith},
  title        = {Pith review of: IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAJMTF2Y}},
  note         = {Machine review of arXiv:2509.01199}
}
read the original abstract

As Industry 4.0 progresses, flexible manufacturing has become a cornerstone of modern industrial systems, with equipment automation playing a pivotal role. However, existing control software for industrial equipment, typically reliant on graphical user interfaces (GUIs) that require human interactions such as mouse clicks or screen touches, poses significant barriers to the adoption of code-based equipment automation. Recently, Large Language Model-based General Computer Control (LLM-GCC) has emerged as a promising approach to automate GUI-based operations. However, industrial settings pose unique challenges, including visually diverse, domain-specific interfaces and mission-critical tasks demanding high precision. This paper introduces IndusGCC, the first dataset and benchmark tailored to LLM-GCC in industrial environments, encompassing 448 real-world tasks across seven domains, from robotic arm control to production line configuration. IndusGCC features multimodal human interaction data with the equipment software, providing robust supervision for GUI-level code generation. Additionally, we propose a novel evaluation framework with functional and structural metrics to assess LLM-generated control scripts. Experimental results on mainstream LLMs demonstrate both the potential of LLM-GCC and the challenges it faces, establishing a strong foundation for future research toward fully automated factories. Our data and code are publicly available at: \href{https://github.com/Golden-Arc/IndustrialLLM}{https://github.com/Golden-Arc/IndustrialLLM.

Figures

Figures reproduced from arXiv: 2509.01199 by the authors.

Figure 1
Figure 1. A chemist controlling an automative synthesis system (on the left of the image, not fully recorded). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Dataset Overview and Statistics. Each subset presents the associated industrial equipment, industrial [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An overview of the three-step procedure introduced in Sections 3.2, 3.3, and 3.4. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An illustration of the evaluation framework. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: An example demonstrating Frame Differencing method [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 45 canonical work pages

  1. [1]

    UnityPro, 2019

    Schneider Electric. UnityPro, 2019. https://www.se.com/us/en/download/ document/UnityPro_EN/[Accessed: May 2025]

    work page 2019

  2. [2]

    IndraWorks, 2019

    Bosch Rexroth. IndraWorks, 2019. https://www.boschrexroth.com/en/us/ products/industrial-solutions/electric-drives-and-controls/ software-tools/[Accessed: May 2025]

    work page 2019

  3. [3]

    TwinCAT, 2011

    Beckhoff Automation. TwinCAT, 2011. https://www. beckhoff.com/en-en/products/automation/twincat/ texxxx-twincat-3-engineering/ [Accessed: May 2025]

    work page 2011

  4. [4]

    Llmind: Orches- trating ai and iot with llm for complex task execution

    Hongwei Cui, Yuyang Du, Qun Yang, Yulin Shao, and Soung Chang Liew. Llmind: Orches- trating ai and iot with llm for complex task execution. IEEE Communications Magazine , 63(4):214–220, 2025

    work page 2025

  5. [5]

    Ll- mind 2.0: Distributed iot automation with natural language m2m communication and lightweight llm agents, 2025

    Yuyang Du, Qun Yang, Liujianfu Wang, Jingqi Lin, Hongwei Cui, and Soung Chang Liew. Ll- mind 2.0: Distributed iot automation with natural language m2m communication and lightweight llm agents, 2025

    work page 2025

  6. [6]

    Chemist-x: Large language model-empowered agent for reaction condition recommendation in chemical synthesis, 2025

    Kexin Chen, Jiamin Lu, Junyou Li, Xiaoran Yang, Yuyang Du, Kunyi Wang, Qiannuan Shi, Jiahui Yu, Lanqing Li, Jiezhong Qiu, Jianzhang Pan, Yi Huang, Qun Fang, Pheng Ann Heng, and Guangyong Chen. Chemist-x: Large language model-empowered agent for reaction condition recommendation in chemical synthesis, 2025

    work page 2025

  7. [7]

    Chemminer: A large language model agent system for chemical literature data mining, 2025

    Kexin Chen, Yuyang Du, Junyou Li, Hanqun Cao, Menghao Guo, Xilin Dang, Lanqing Li, Jiezhong Qiu, Pheng Ann Heng, and Guangyong Chen. Chemminer: A large language model agent system for chemical literature data mining, 2025

    work page 2025

  8. [8]

    Autowebglm: A large language model-based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD ’24, page 5295–5306, New York, NY , USA,

    work page

Show all 52 references
  1. [9]

    Webcanvas: Benchmarking web agents in online environments, 2024

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. Webcanvas: Benchmarking web agents in online environments, 2024

    2024

  2. [10]

    Weblinx: real-world website navigation with multi-turn dialogue

    Xing Han Lù, Zdenˇek Kasner, and Siva Reddy. Weblinx: real-world website navigation with multi-turn dialogue. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  3. [11]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, v...

    2022

  4. [12]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of...

    2024

  5. [13]

    Mind2web: towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: towards a generalist agent for the web. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY , USA, 2023. Cu...

    2023

  6. [14]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. CoRR, abs/2401.16158, 2024. 10

    2024 arXiv

  7. [15]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, ...

    2024

  8. [16]

    Spa-bench: A comprehensive benchmark for smartphone agent evaluation, 2025

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, and Kun Shao. Spa-bench: A comprehensive benchmark for smartphone agent evaluation, 2025

    2025

  9. [17]

    An- droidinthewild: A large-scale dataset for android device control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing ...

    2023

  10. [18]

    AndroidLab: Training and systematic benchmarking of android autonomous agents

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. AndroidLab: Training and systematic benchmarking of android autonomous agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, ed...

    2025

  11. [19]

    Androidworld: A dynamic benchmarking environment for autonomous agents, 2025

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchma...

    2025

  12. [20]

    macosworld: A multilingual interactive benchmark for gui agents, 2025

    Pei Yang, Hai Ci, and Mike Zheng Shou. macosworld: A multilingual interactive benchmark for gui agents, 2025

    2025

  13. [21]

    Windows agent arena: Evaluating multi-modal os agents at scale, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024

    2024

  14. [22]

    Worldgui: An interactive benchmark for desktop gui automation from any starting point, 2025

    Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, and Mike Zheng Shou. Worldgui: An interactive benchmark for desktop gui automation from any starting point, 2025

    2025

  15. [23]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for op...

    2024

  16. [24]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Ru...

    2024

  17. [25]

    Agentboard: an analytical evaluation board of multi-turn llm agents

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: an analytical evaluation board of multi-turn llm agents. In Proceedings of the 38th International Conference on Neural Information Processing Syste...

    2025

  18. [26]

    Gaia: a benchmark for general ai assistants, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023

    2023

  19. [27]

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor 11 Zhong, Lu Chen, Kai...

    2025

  20. [28]

    Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2025

    Lawrence Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2025

    2025

  21. [29]

    Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks

    Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. In A. Globerson,...

    2024

  22. [30]

    Webvln: vision-and-language navigation on websites

    Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, and Qi Wu. Webvln: vision-and-language navigation on websites. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Art...

    2024

  23. [31]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    2023

  24. [32]

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empir...

    2024

  25. [33]

    Mmina: Benchmarking multihop multimodal internet agents, 2025

    Shulin Tian, Ziniu Zhang, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents, 2025

    2025

  26. [34]

    VisualWebArena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,...

    2024

  27. [35]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024

    2024

  28. [36]

    Moveit! task constructor for task-level motion planning

    Michael Görner, Robert Haschke, Helge Ritter, and Jianwei Zhang. Moveit! task constructor for task-level motion planning. In 2019 International Conference on Robotics and Automation (ICRA), pages 190–196, 2019

    2019

  29. [37]

    GNU Radio, 2023

    GNU Radio. GNU Radio, 2023. https://www.gnuradio.org/[Accessed: May 2025]

    2023

  30. [38]

    USRP Hardware Driver, 2018

    Ettus Research. USRP Hardware Driver, 2018. https://github.com/ EttusResearch/uhd[Accessed: May 2025]

    2018

  31. [39]

    Robot op- erating system 2: Design, architecture, and uses in the wild

    Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. Robot op- erating system 2: Design, architecture, and uses in the wild. Science Robotics, 7(66):eabm6074, 2022

    2022

  32. [40]

    Gazebo, 2024

    Open Robotics Foundation. Gazebo, 2024. https://gazebosim.org/[Accessed: May 2025]

    2024

  33. [41]

    Wireshark, 2024

    Wireshark Foundation. Wireshark, 2024. https://www.wireshark.org/[Accessed: May 2025]. 12

    2024

  34. [42]

    Smith and M.S

    T.F. Smith and M.S. Waterman. Identification of common molecular subsequences. Journal of Molecular Biology, 147(1):195–197, 1981

    1981

  35. [43]

    Structural function based code clone detection using a new hybrid technique

    Yanming Yang, Zhilei Ren, Xin Chen, and He Jiang. Structural function based code clone detection using a new hybrid technique. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), volume 01, pages 286–291, 2018

    2018

  36. [44]

    Omer and Asma’a Y

    Shahad S. Omer and Asma’a Y . Hammo. A comparative study for language independent code clone detection. AIP Conference Proceedings, 3211(1):030050, 05 2025

    2025

  37. [45]

    PyAutoGUI, 2019

    Al Sweigart. PyAutoGUI, 2019. https://github.com/asweigart/ pyautogui[Accessed: May 2025]

    2019

  38. [46]

    Dutta, A

    A. Dutta, A. Gupta, and A. Zissermann. VGG image annotator (VIA). http://www.robots.ox.ac.uk/ vgg/software/via/, 2016. Version: X.Y .Z, Accessed: IN- SERT_DATE_HERE

    2016

  39. [47]

    trigger-and-feedback

    Abhishek Dutta and Andrew Zisserman. The VIA annotation software for images, audio and video. In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, New York, NY , USA, 2019. ACM. 13 A Algorithm details of Evaluation Framework Visual Observation Phase I...

    2019

  40. [49]

    Full-Screen Video: Recorded at a resolution of 1920×1080 and a frame rate of 60 FPS, utilizing FFmpeg for efficient real-time encoding

    1920

  41. [50]

    Mouse Movement Trajectory: Captured as tuples of (timestamp, x-coordinate, y-coordinate) for each frame, ensuring sub-frame temporal resolution

  42. [51]

    Mouse Events: Logged with timestamp, x-coordinate, y-coordinate, button type (e.g., left/right), and button state (e.g., pressed/released)

  43. [52]

    Keyboard Events: Recorded with timestamp, key identifier, and key states (e.g., pressed/released). The script is designed for stable operation on commonly used operating systems where equipment control software is deployed (such as the latest Ubuntu Release and Windows 10/11) ...

    1920

  44. [2024]

    Association for Computing Machinery

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.