Pith. sign in

REVIEW 3 major objections 41 references

A single GUI demonstration can become a reusable CLI skill that replays without live model inference, using layered visual localization and post-action checks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 20:24 UTC pith:FIT6WNWM

load-bearing objection Solid, modest systems paper: demo-to-OpenClaw GUI skills without runtime LLM, reliable on one machine, evaluation thin but claims stay inside that box. the 3 major comments →

arxiv 2606.05171 v1 pith:FIT6WNWM submitted 2026-04-15 cs.HC cs.SE

AppAgent-Claw: CLI Is All You Need for GUI Automation

Zhixue Song , Zhiheng Zhang , Yi Song , Chi Zhang This is my paper
classification cs.HC cs.SE
keywords GUI automationdemonstration-driven automationskill reusetemplate matchingcommand-line agentsworkflow replayvisual localizationRPA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many real software tasks still live only behind graphical interfaces with no stable API, so an agent that plans well can still be blocked from acting. Live language-model GUI agents can operate those screens in open-ended fashion, but repeated model calls make them too slow, costly, and inconsistent to serve as lightweight everyday skills. This paper presents a demonstration-driven system that records a foreground walkthrough once, packages it with rich visual and window context, and replays it many times as a callable skill with no runtime model inference. Reliability is produced by layered target finding—local anchor match, then broader context match, then monitor-relative coordinates—plus validation that the intended on-screen effect actually occurred. In the same-machine setting the authors target, every reported baseline and perturbed replay completed successfully.

Core claim

GUI-bound tasks can be converted into efficient, reusable automation skills by recording one user demonstration with rich visual and window metadata, then replaying through layered localization and validation-coupled execution without any live model inference. Under the intended same-machine regime, the reported trials—fifty baseline runs and thirty-six runs under dark mode, zoom, and different initial UI state—all finished with full end-to-end and per-step success.

What carries the argument

The record-once, annotate-once, replay-many pipeline, carried by a three-layer localization policy (anchor crop in the recorded local search region, then broader context match on the monitor, then constrained relative-coordinate fallback) coupled with post-action validation that treats a confirmed on-screen effect—not a dispatched action—as success.

Load-bearing premise

Replay stays reliable only when the same machine keeps a similar monitor layout, window geometry, and foreground UI state so the recorded visual anchors and relative coordinates still land on the right controls.

What would settle it

Re-run the five benchmark workflows after a non-trivial redesign of one app’s UI or after moving the recorded windows onto a different monitor arrangement; if end-to-end success collapses even with all three localization layers enabled, the claim of practical reusable replay does not hold outside near-identical conditions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • GUI tasks without APIs can be packaged as fast callable skills instead of invoking a full live GUI agent on every run.
  • Failures become diagnosable through structured resolve attempts and validation outcomes rather than opaque model errors.
  • Modest visual drift such as dark mode, zoom, or a different initial state can be absorbed by fallback layers without rewriting the skill.
  • Reviewer-confirmed parameterization of free-form text lets one recorded flow serve messaging-style tasks with different content.
  • Demonstration-driven skills can act as a workflow substrate that broader open-domain GUI agents later learn from and invoke.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agent skill libraries may grow faster from users recording daily apps than from training ever-larger open-world controllers for every interface.
  • The same preserved-context plus layered-fallback pattern could harden classical RPA tools that still depend on brittle fixed selectors.
  • A practical hybrid upgrade path would re-annotate or re-record only when Layer-3 relative fallback is hit too often under routine use.
  • Because dark mode forced almost all resolves onto coordinate fallback, theme-aware anchor banks look like a natural next engineering step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. AppAgent-Claw is a demonstration-driven GUI automation skill for the OpenClaw platform. It converts a single user walkthrough into a reusable CLI-invokable skill via a record–annotate–replay pipeline that stores rich visual and window context, then localizes targets with a three-layer policy (local anchor match → broader context match → monitor-relative coordinates) and advances only after post-action validation of on-screen effects. The design deliberately avoids runtime LLM inference so that GUI-bound tasks can meet OpenClaw’s requirements of fast, consistent, lightweight skills. Experiments on one macOS machine report 50/50 baseline and 36/36 perturbed end-to-end successes across five hand-chosen workflows (Tables 9–10), with Layer 2/3 absorbing a non-trivial fraction of resolves, especially under dark mode.

Significance. If the reliability claim holds under the stated same-machine regime, the work offers a practical bridge between brittle RPA-style replay and expensive open-world GUI agents: a way to package repeated foreground GUI workflows as OpenClaw skills without per-invocation model cost. Strengths include a carefully specified data protocol and action space (§3.3–3.4), explicit layered safeguards and observability (§3.7–3.9), an open project page/codebase, and an honest Discussion (§5) that bounds the operating assumptions rather than overclaiming open-domain autonomy. The contribution is systems- and engineering-oriented rather than a new learning method; its value depends on whether the evaluation can show that the layered design, not a carefully matched suite, is what produces the reported reliability.

major comments (3)
  1. §4.2–4.3 and Tables 9–10 report 100% E2E and per-step success (86 runs, 366 click-like resolves) but include no comparison baselines. Without at least coordinate-only replay, pure single-layer template matching, and/or a lightweight LLM GUI agent on the same workflows, it is not possible to attribute the perfect success rates to the layered resolver and validation model rather than to a non-stressing evaluation suite. This comparison is load-bearing for the claim that AppAgent-Claw is a practical, reliable skill rather than a carefully matched demo.
  2. §4.2–4.3 explicitly state that L1–L3 hit rates are attribution statistics over successful resolver calls, not removal-based ablations. The paper’s central design claim is that layered localization and validation-coupled execution produce graceful degradation. Without ablations that disable Layer 2, Layer 3, and/or posterior validation (and report resulting failures or retries), the 14.7% non-L1 share and the dark-mode shift to Layer 3 remain descriptive rather than causal evidence for the architecture.
  3. The evaluation uses five hand-chosen macOS workflows on a single machine, with only three runs per perturbed condition (§4.1, §4.3). The Discussion (§5) correctly restricts claims to same-machine, similar-layout, foreground use, yet the abstract and conclusion still present the system as a general practical solution for integrating GUI-bound tasks into OpenClaw. Either the experimental suite must be broadened (more apps, controlled failure cases, multi-monitor or layout drift that actually breaks anchors) or the claims must be tightened to match the demonstrated regime so that 100% success is not over-read as general skill reliability.

Circularity Check

0 steps flagged

No circularity: empirical systems paper whose success metrics are external on-screen outcomes, not quantities forced by definition or fit.

full rationale

AppAgent-Claw is a demonstration-driven engineering system (record-annotate-replay) evaluated by end-to-end and per-step run completion under same-machine conditions (Tables 9–10). There is no derivation chain in which a claimed prediction or first-principles result reduces to its inputs by construction. Match thresholds (0.92), crop sizes, wait timers, and validation modes are design parameters, not fitted quantities re-presented as predictions. Layer hit rates are explicitly attribution statistics over successful resolves, not removal ablations or circular forecasts. Self-citations to AppAgent/AppAgentX appear only as related-work context for open-world GUI agents and do not supply a uniqueness theorem or load-bearing premise that forces the present design. Success is defined by observed on-screen effects and completed runs, which are external and falsifiable within the stated operating regime. Experimental narrowness (same machine, five workflows, no baselines) is a validity concern, not circularity. Score 0; steps empty.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing content is mostly engineering design choices and operating-regime assumptions, not free physical constants or new particles. Reliability claims rest on hand-chosen matching thresholds, crop geometry, same-machine visual similarity, and accessibility APIs when available. No formal derivation; the ledger is dominated by domain assumptions and tunable parameters that define when replay is expected to work.

free parameters (5)
  • template_match_threshold
    Default 0.92 for OpenCV matchTemplate; directly gates Layer 1/2 success vs. fallback.
  • anchor_and_context_crop_sizes
    Defaults 96×96 anchor, 240×160 context, search_padding 96 px; define what visual evidence is preserved for replay.
  • event_aggregation_thresholds
    wait 0.8s, double-click 0.35s/4px, long-press 0.5s/8px, IME commit delays; determine how raw events become structured steps.
  • low_information_template_filters
    Grayscale std ≤ 4.0 and RGB dynamic range ≤ 16 skip templates; hand-set quality gate.
  • validation_poll_window
    Default 0.2s poll / 2.0s window (1.0s fixed wait when AX text unavailable); defines when an action is declared successful.
axioms (5)
  • domain assumption Same-machine, similar monitor/window layout and foreground UI state make recorded anchors and monitor-relative coordinates informative at replay.
    Stated operating regime in §4.1 and Discussion §5; without it the localization stack has no grounding.
  • domain assumption OpenCV template matching (TM_CCOEFF_NORMED or TM_SQDIFF_NORMED) plus relative-coordinate fallback is sufficient for the targeted repeated workflows.
    Core of §3.7; no learned detector is used at replay.
  • domain assumption Post-action visual or accessibility checks (anchor present/absent, focused text) are a valid success criterion distinct from action dispatch.
    Validation-coupled execution model in §3.8.
  • domain assumption macOS accessibility APIs (AXFocusedUIElement / AXValue) when available improve text capture and validation; otherwise coarser key/wait behavior is acceptable.
    Recording and type_text validation paths in §3.5 and §3.8.
  • ad hoc to paper Reviewers must not edit the recorded action sequence; wrong capture requires re-recording.
    Annotation policy in §3.6; protects replay fidelity but constrains repair.
invented entities (2)
  • AppAgent-Claw skill package (flow definition + assets + layered resolver + validation executor) independent evidence
    purpose: Package a demonstrated GUI workflow as a lightweight, inference-free OpenClaw skill.
    The system is the paper’s primary artifact; independent evidence is the reported replay experiments and the linked repository, not external theory.
  • Three-layer localization policy (local anchor → monitor context → relative coordinates) no independent evidence
    purpose: Resolve click targets under mild visual drift without live model inference.
    Design construct of §3.7; evidence is layer-hit statistics, not an external physical entity.

pith-pipeline@v1.1.0-grok45 · 17021 in / 3250 out tokens · 29449 ms · 2026-07-12T20:24:02.053897+00:00 · methodology

0 comments
read the original abstract

The OpenClaw platform provides a practical foundation for automation through its skill-oriented architecture, organizing external capabilities into lightweight, reusable components that can be invoked efficiently through a command-line interface (CLI). However, a significant bottleneck remains: many real-world tasks are confined to graphical user interfaces (GUIs) with no stable API available. While LLM-based GUI agents offer generality, their reliance on repeated live model inference makes them too slow, costly, and inconsistent to serve as efficient OpenClaw skills. In this paper, we present AppAgent-Claw, a demonstration-driven system that converts GUI workflows into reliable, reusable skills without runtime inference. By following a ``record-once, replay-many'' paradigm, the system captures rich contextual metadata to facilitate robust execution. It employs a layered localization strategy to handle visual shifts and a validation-coupled execution model to ensure intended on-screen effects. AppAgent-Claw provides a practical, efficient, and diagnosable solution for integrating GUI-bound tasks into the OpenClaw ecosystem.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 5 canonical work pages

  1. [1]

    Openclaw.https://github.com/openclaw/openclaw, 2026

    OpenClaw. Openclaw.https://github.com/openclaw/openclaw, 2026. GitHub repository; accessed 2026-04-14

    2026

  2. [2]

    AppAgent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. AppAgent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 70:1–70:20. Association for Computing Machinery, 2025. doi: 10.1145/3706598.3713600

    doi:10.1145/3706598.3713600 2025

  3. [3]

    AppAgentX: Evolving GUI agents as proficient smartphone users, 2025

    Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, Joey Tianyi Zhou, and Chi Zhang. AppAgentX: Evolving GUI agents as proficient smartphone users, 2025. URLhttps://arxiv.org/abs/2503.02268

    Pith/arXiv arXiv 2025

  4. [4]

    Robotic process automation.Business & Information Systems Engineering, 60(4):269–272, 2018

    Wil MP van der Aalst, Martin Bichler, and Armin Heinzl. Robotic process automation.Business & Information Systems Engineering, 60(4):269–272, 2018. doi: 10.1007/s12599-018-0542-4

    work page doi:10.1007/s12599-018-0542-4 2018

  5. [5]

    Leemans, Chun Ouyang, Arthur H.M

    Rehan Syed, Suriadi Suriadi, Michael Adams, Wasana Bandara, Sander J.J. Leemans, Chun Ouyang, Arthur H.M. ter Hofstede, Inge van de Weerd, Moe Thandar Wynn, and Hajo A. Reijers. Robotic process automation: Contemporary themes and challenges.Computers in Industry, 115:103162, 2020. doi: 10.1016/j.compind.2019. 103162

    work page doi:10.1016/j.compind.2019 2020

  6. [6]

    Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. Sikuli: Using GUI screenshots for search and automation. InProceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, pages 183–192,

  7. [7]

    doi: 10.1145/1622176.1622213

    doi:10.1145/1622176.1622213

  8. [8]

    Mobile- agent: Autonomous multi-modal mobile device agent with visual perception, 2024

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile- agent: Autonomous multi-modal mobile device agent with visual perception, 2024. URLhttps://arxiv.org/abs/ 2401.16158. Accepted by the ICLR 2024 Workshop on Large Language Model Agents

    Pith/arXiv arXiv 2024

  9. [9]

    AutoDroid: LLM-powered task automation in android, 2024

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. AutoDroid: LLM-powered task automation in android, 2024. URL https: //arxiv.org/abs/2308.15272. Published in MobiCom 2024

    Pith/arXiv arXiv 2024

  10. [10]

    WebVoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6864–6890. Association for Computational Linguistics, 2024....

    doi:10.18653/v1/2024.acl-long.371 2024

  11. [11]

    AutoWebGLM: A large language model-based web navigating agent, 2024

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. AutoWebGLM: A large language model-based web navigating agent, 2024. URLhttps://arxiv.org/abs/2404.03648. Accepted to KDD 2024

    Pith/arXiv arXiv 2024

  12. [12]

    OpenWebVoyager: Building multimodal web agents via iterative real-world exploration, feedback and 11 optimization

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. OpenWebVoyager: Building multimodal web agents via iterative real-world exploration, feedback and 11 optimization. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27545–27564. A...

    doi:10.18653/v1/2025 2025

  13. [13]

    AssistGUI: Task-oriented PC graphical user interface automation

    Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, and Mike Zheng Shou. AssistGUI: Task-oriented PC graphical user interface automation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13289–13298, 2024. URLht...

    2024

  14. [14]

    UFO: A UI-focused agent for Windows OS interaction

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. UFO: A UI-focused agent for Windows OS interaction. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume...

    2025

  15. [15]

    URLhttps://aclanthology.org/2025.naacl-long.26/

    doi: 10.18653/v1/2025.naacl-long.26. URLhttps://aclanthology.org/2025.naacl-long.26/

    doi:10.18653/v1/2025.naacl-long.26 2025

  16. [16]

    OS-copilot: Towards generalist computer agents with self-improvement, 2024

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. OS-copilot: Towards generalist computer agents with self-improvement, 2024. URLhttps: //arxiv.org/abs/2402.07456

    Pith/arXiv arXiv 2024

  17. [17]

    Screenagent: A vision language model-driven computer control agent

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 6433–6441, 2024. doi: 10.24963/ijcai.2024/711. URLhttps://www.ijcai.org/proceedings/2024/711

    doi:10.24963/ijcai.2024/711 2024

  18. [18]

    Agent S: An Open Agentic Framework that Uses Computers Like a Human

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent S: An Open Agentic Framework that Uses Computers Like a Human. InInternational Conference on Learning Representations (ICLR),

  19. [19]

    URLhttps://openreview.net/forum?id=lIVRgt4nLv

  20. [20]

    Agent S2: A compositional generalist-specialist framework for computer use agents

    Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent S2: A compositional generalist-specialist framework for computer use agents. InConference on Language Modeling (COLM), 2025. URLhttps://openreview.net/forum?id=zg5is4GJ3R

    2025

  21. [21]

    Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu

    Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradl...

    Pith/arXiv arXiv 2024

  22. [22]

    The unreasonable effectiveness of scaling agents for computer use, 2025

    Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, and Xin Eric Wang. The unreasonable effectiveness of scaling agents for computer use, 2025. URLhttps://arxiv.org/abs/2510.02250

    arXiv 2025

  23. [23]

    SeeClick: Harnessing GUI grounding for advanced visual GUI agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332. Association for Computational Linguistics, 2024. doi: 10.18653/v...

    doi:10.18653/v1/2024.acl-long.505 2024

  24. [24]

    CogAgent: A visual language model for GUI agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: A visual language model for GUI agents. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281– 14290, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/ht...

    2024

  25. [25]

    Rico: A mobile app dataset for building data-driven design applications

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. InProceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pages 845–854, 2017. doi: 10.1145/3126594.3126651

    doi:10.1145/3126594.3126651 2017

  26. [26]

    Mapping natural language instructions to mobile UI action sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8198–8210. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.729. URLhttps://aclanthology.org/202...

    doi:10.18653/v1/2020.acl-main.729 2020

  27. [27]

    Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap

    Peter C. Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. InProceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, ...

    2022

  28. [28]

    A zero-shot language agent for computer control with structured reflection

    Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A zero-shot language agent for computer control with structured reflection. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 11261–11274. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.753. URL https://aclanthology.org/2023.findings-...

    doi:10.18653/v1/2023.findings-emnlp.753 2023

  29. [29]

    Witten, and Kenneth A

    David Maulsby, Ian H. Witten, and Kenneth A. Kittlitz. Metamouse: Specifying graphical procedures by example. InProceedings of the 16th Annual Conference on Computer Graphics and Interactive Techniques, pages 127–136,

  30. [30]

    doi: 10.1145/74333.74346

    work page doi:10.1145/74333.74346

  31. [31]

    EAGER: Programming repetitive tasks by example

    Allen Cypher. EAGER: Programming repetitive tasks by example. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 33–39, 1991. doi: 10.1145/108844.108850

    work page doi:10.1145/108844.108850 1991

  32. [32]

    Brad A. Myers. Scripting graphical applications by demonstration. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 534–541, 1998. doi: 10.1145/274644.274716

    work page doi:10.1145/274644.274716 1998

  33. [33]

    MIT Press, 1993

    Allen Cypher, editor.Watch What I Do: Programming by Demonstration. MIT Press, 1993

    1993

  34. [34]

    Morgan Kaufmann, 2001

    Henry Lieberman, editor.Your Wish Is My Command: Programming by Example. Morgan Kaufmann, 2001

    2001

  35. [35]

    Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. SUGILITE: Creating multimodal smartphone automation by demonstration. InProceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 6038–6049, 2017. doi: 10.1145/3025453.3025483

    doi:10.1145/3025453.3025483 2017

  36. [36]

    Amant, Henry Lieberman, Richard Potter, and Luke S

    Robert St. Amant, Henry Lieberman, Richard Potter, and Luke S. Zettlemoyer. Programming by example: Visual generalization in programming by example.Communications of the ACM, 43(3):107–114, 2000

    2000

  37. [37]

    Imitation learning: A survey of learning methods.ACM Computing Surveys (CSUR), 50(2):1–35, 2017

    Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods.ACM Computing Surveys (CSUR), 50(2):1–35, 2017. doi: 10.1145/3054912

    doi:10.1145/3054912 2017

  38. [38]

    URLhttps://developer.apple.com/library/archive/ documentation/Accessibility/Conceptual/AccessibilityMacOSX/

    Apple Inc.Accessibility Programming Guide for OS X, 2015. URLhttps://developer.apple.com/library/archive/ documentation/Accessibility/Conceptual/AccessibilityMacOSX/. Documentation archive; updated 2015-04-08; accessed 2026-04-13

    2015

  39. [39]

    URLhttps://developer.apple.com/documentation/ applicationservices/axvalue

    Apple Inc.AXValue | Apple Developer Documentation, n.d. URLhttps://developer.apple.com/documentation/ applicationservices/axvalue. Apple Developer Documentation; accessed 2026-04-13

    2026

  40. [40]

    The OpenCV library.Dr

    Gary Bradski. The OpenCV library.Dr. Dobb’s Journal of Software Tools, 25(11):120–125, 2000

    2000

  41. [41]

    John Wiley & Sons, 2009

    Roberto Brunelli.Template matching techniques in computer vision: theory and practice. John Wiley & Sons, 2009. 13

    2009