Pith. sign in

REVIEW 3 major objections 3 minor 34 references

This paper argues that desktop computer-use agents fail less from lack of model intelligence and more from a brittle motor interface, and introduces Tactile, an open tool layer that gives agents semantic, verifiable 'hands and feet' for des

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:04 UTC pith:HDGMSGM3

load-bearing objection A genuinely useful systems integration paper whose headline effect is real but whose cross-agent numbers are an oracle upper bound — worth refereeing, but only after a fixed-policy comparison is reported. the 3 major comments →

arxiv 2607.14443 v1 pith:HDGMSGM3 submitted 2026-07-16 cs.AI

Tactile: Giving Computer-Using Agents Hands and Feet

Yong Liu , Zhenyi Zhong , Zhanpeng Shi This is my paper
classification cs.AI
keywords computer-use agentsdesktop automationaccessibility APIGUI groundingaction-grounded interface stateobserve-ground-act-verifytool layerSuccess@100
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that desktop computer-use agents fail less because of model intelligence and more because their interface to software is a brittle motor layer: screenshot, predict coordinates, click, hope. It introduces Tactile, an open tool layer that turns UI evidence (accessibility semantics, OCR text, visual regions) into action-grounded interface states—candidates with roles, state, geometry, affordances, and verification cues—and runs an observe-ground-act-verify loop that prefers native semantic actions. On macOSWorld-style tasks, adding Tactile raises Success@100 from 41.1% to 50.0% overall, with larger gains on accessibility-adapted tasks, and a 96-task cross-agent subset shows gains across four agents. The paper's central claim is that a reusable execution substrate—exposing actions as semantic, verifiable, auditable objects—is a necessary complement to stronger models.

Core claim

The central discovery is that separating target grounding, action execution, and outcome verification—and preferring accessibility semantics before OCR coordinates before visual fallback—improves grounded desktop operation across agents. Tactile compiles heterogeneous UI evidence into compact target candidates: each candidate carries source labels, role or text, state, geometry, executable affordances, and verification cues; agents choose candidates, and the runtime executes the safest primitive and re-observes to verify. The authors report that this raises Codex Success@100 from 41.1% to 50.0% overall, from 45.2% to 55.3% on accessibility-adapted tasks, and produces consistent gains on a 96

What carries the argument

The action-grounded interface state is the central object: a ranked set of target candidates built from accessibility elements, OCR lines, and visual regions, each with source labels, role/text, state, geometry, executable affordances, and provenance. The accessibility-first operating ladder orders evidence—Level 1 native accessibility semantics, Level 2 OCR-grounded coordinates, Level 3 visual fallback—so each step uses the richest available signal. The observe-ground-act-verify loop separates the four decisions that screenshot-first control collapses: collect evidence, select a candidate, execute the safest primitive (semantic action when available, coordinate click otherwise), and re-obse

Load-bearing premise

The reported gains are attributed to the semantic operating layer, but the cross-agent comparison uses a best-of-three upper bound with no uncertainty estimates, so the improvement could shrink if an agent must commit to a single policy in advance.

What would settle it

Re-run the 96-task cross-agent benchmark with a single pre-committed 'always use Tactile' policy and report confidence intervals; if the Codex gain collapses below noise or another agent's gain inverts, the claim of consistent gains from the semantic substrate would fail. Also verify whether the main Codex graded comparison uses the same upper-bound rule.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adding Tactile raises Codex Success@100 from 41.1% to 50.0% overall and from 45.2% to 55.3% on accessibility-adapted tasks.
  • Gains generalize across four different agents on a 96-task subset, with the largest improvements on tasks where applications expose useful accessibility metadata.
  • Accessibility semantics provide the strongest action and verification contracts; OCR and visual fallback remain necessary for semantically opaque interfaces.
  • Separating grounding from execution makes failures attributable: traces can show whether a target was absent, filtered, mis-ranked, mis-executed, or unverifiable.
  • Because the same accessibility metadata serves screen-reader users and agents, improving human accessibility also improves agent operability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The cross-agent 'With Tactile' score is a best-of-several-settings upper bound compared with a single no-skill baseline; a deployed agent that must commit to one policy in advance could see smaller, or in the weakest case negligible, gains.
  • A direct test of the mechanism would compare Tactile against a control that normalizes coordinates and adds verification without semantic targets; if that control matches Tactile's gains, the accessibility-first ladder is not the active ingredient.
  • The largest untested upside is policy learning: a router that decides per-step whether to invoke semantic, OCR, or visual tools could beat both the no-skill and always-Tactile baselines.
  • Because the implementation is strongest on macOS with only early Windows support, the cross-platform claim is a design commitment, not yet an empirical result.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces TACTILE, an open-source MCP-compatible tool layer for macOS desktop agents. It converts accessibility-tree data, OCR text, and visual regions into a compact, action-grounded interface state, and wraps execution in an observe-ground-act-verify loop that prefers native semantic actions over coordinate clicks. The authors evaluate on macOSWorld-style tasks, reporting Success@100 gains for Codex (41.1% to 50.0% overall; 45.2% to 55.3% on accessibility-adapted tasks) and for a 96-task cross-agent subset across Codex, Claude Code, OpenCode, and Goose. The paper is candid that the cross-agent 'With Tactile' score is a skill-optional upper bound, and §5.3 and §7 list limitations including missing uncertainty estimates and the need for stronger baselines.

Significance. If the central causal claim is established, TACTILE would be a useful reusable execution substrate: it makes targetability, actionability, verifiability, and auditability first-class properties, and the accessibility-first ladder is a sensible design principle. The paper has real strengths: it is open-source, evaluated against an external benchmark, includes trace-based examples, and is unusually candid about its limitations. However, the current evidence does not yet support the strength of the abstract's claims because the cross-agent comparison uses an oracle-style upper bound and the main results lack uncertainty quantification.

major comments (3)
  1. [§5, cross-agent comparisons] The abstract and §5.1 present 'consistent gains across Codex, Claude Code, OpenCode, and Goose' as if they describe a deployable policy. But §5 states: 'The reported cross-agent With Tactile score is a skill-optional upper bound computed from the best per-task result among no-skill, tactile-implicit, and tactile-explicit settings.' Comparing this best-of-three score against a single no-skill baseline inflates the apparent gain by construction, because a real agent must commit to one policy before seeing the task. Please report a fixed-policy comparison and state explicitly whether the headline Codex numbers in §5.1 also use the upper-bound rule; if they do, the abstract's lead numbers are oracle upper bounds and must be labeled as such.
  2. [§5.1 / §5.3] No uncertainty estimates are reported anywhere. On the 96-task cross-agent subset, the Goose gain is +2.08 percentage points, which is approximately two tasks; without confidence intervals, a paired per-task analysis, or multiple seeds, the 'consistent gains' claim is not robustly supported. §5.3 defers uncertainty to future work, but the abstract states the improvements as definitive. At minimum, report Wilson intervals or bootstrap CIs for each Success@100 estimate and for the deltas, and state the number of tasks in each split.
  3. [§5, task splits] The partition into AX-adapted and Limited-AX tasks is introduced post hoc and is central to the mechanistic interpretation that accessibility semantics drive the gain. No independent criteria, annotation protocol, or reliability measure are given for this split. Because the headline effect is largest on the AX-adapted subset, the split should be specified before evaluating, or at least independently validated; otherwise the interpretation risks being shaped by the authors' own categorization of the same tasks used in the comparison.
minor comments (3)
  1. [§4.2] There is an unresolved 'Figure??' reference; the evidence-compiler figure is missing or its cross-reference is broken.
  2. [Abstract / §5.1] The abstract rounds 41.06% to 41.1% and 50.00% to 50.0%; this is fine, but the abstract omits the upper-bound caveat that the body applies to the cross-agent comparison. A one-sentence qualifier ('under a skill-optional upper-bound evaluation') would prevent over-reading.
  3. [§5.2] The Zoom trace example is informative, but no success/failure was reported for that specific trace. Stating whether the recorded trajectory completed the task would strengthen the illustrative value.

Circularity Check

1 steps flagged

Cross-agent 'consistent gains' reduce by construction: the With Tactile score is an oracle max that includes the no-skill baseline.

specific steps
  1. fitted input called prediction [§5 Results, first paragraph; §5.1 Main Results (cross-agent paragraph); Abstract]
    "The reported cross-agent With Tactile score is a skill-optional upper bound computed from the best per-task result among no-skill, tactile-implicit, and tactile-explicit settings; it estimates the value of making TACTILE available, not a learned routing policy."

    The candidate set used for the 'With Tactile' score includes the no-skill baseline itself, so the per-task maximum is, by construction, at least the no-skill score on every task. The reported 'gains' (e.g., 'total success improves from 38.54% to 50.00% for Codex ... 41.67% to 43.75% for Goose') and the abstract's claim of 'consistent gains across Codex, Claude Code, OpenCode, and Goose' are therefore mathematically guaranteed by the max operation, not empirical evidence that TACTILE adds value. The per-task best setting is fitted to the test data and then presented as the treatment condition's success rate, which is the fitted-input-called-prediction pattern.

full rationale

The paper is a systems-and-evaluation paper, not a formal derivation, and most of its content (tool design, MCP interface, coordinate contracts) is not circular. There are no load-bearing self-citations: the only external code acknowledgment (mediar-ai) is an implementation dependency, not a result used to justify the central claim. The main circularity is confined to the cross-agent evaluation protocol. The 'With Tactile' condition is an oracle upper bound that includes the baseline; hence the cross-agent 'consistent gains' headline is forced by construction and cannot serve as evidence for TACTILE's causal benefit. The paper is transparent about this in §5.3, noting the score is 'an upper bound rather than a learned routing policy,' but the abstract presents the cross-agent improvement as an empirical result without that caveat. The main Codex comparison (41.06% to 50.00%) is stated separately, and the paper does not explicitly say whether it uses the same upper-bound rule; if it is a fixed-policy comparison, that part is not circular, though its statistical support is thin and no uncertainty is reported. The AX-adapted task split is post hoc but not definitionally circular: it groups tasks by accessibility-metadata availability and then observes a larger gain there, which is a mechanism-consistent correlation rather than a result equivalent to its inputs. Overall, one of the paper's two headline claims (cross-agent consistent gains) reduces by construction, so a score of 6 is appropriate; the central Codex result may remain independently meaningful, preventing a higher score.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

No physical constants or fitted physics quantities appear; this is an empirical systems paper. The hand-chosen elements are the candidate ranking weights, the post-hoc AX/Limited partition, and the best-of-three upper-bound selector, all of which shape the reported gains. The load-bearing assumptions are domain assumptions about accessibility fidelity, OCR reliability, baseline representativeness, and the validity of the macOSWorld subset. One software abstraction is introduced, and it carries a falsifiable handle via the open-source repository.

free parameters (3)
  • candidate ranking weights
    Algorithm 1's ScoreByActionabilityAndStability ranks candidates by visibility, enabled state, name quality, focus, action support, textual relevance, and recency; the weights are not specified and are hand-tuned in the implementation. Ranking determines which candidates the model sees, so it affects the observed gains.
  • AX-adapted vs Limited-AX task partition = post-hoc categorical split; per-group counts not reported
    §5 partitions tasks by accessibility quality after the fact. The larger gain on AX-adapted tasks is the main mechanism evidence, but the split is not pre-registered and the per-group N is not given, so the split's stability is unknown.
  • skill-optional upper-bound selector = max over {no-skill, implicit, explicit} per task
    §5: cross-agent With-Tactile scores are the best per-task result among three conditions, chosen after the fact. This hand-chosen selection rule inflates reported gains relative to the single-condition baseline and is a free evaluation choice the central cross-agent claim depends on.
axioms (4)
  • domain assumption macOS Accessibility API exposes faithful, actionable UI semantics (roles, names, states, actions) for AX-adapted tasks
    Level 1 of the ladder (§3.2). §7 acknowledges accessibility metadata can be missing, incorrect, or stale [18]; the mechanistic interpretation of larger gains on AX-adapted tasks depends on this premise.
  • domain assumption OCR text positions and normalized coordinates map reliably to interactive targets
    Level 2 of the ladder (§3.2). §7 lists OCR failure modes (low contrast, icons, animations, unsupported languages), so the fallback path is assumed accurate enough to contribute to the gains.
  • domain assumption Success@100 on the currently graded macOSWorld sample set is a valid measure of computer-use agent capability
    §5 adopts Success@100 on 'currently graded' samples; the representativeness of this subset is not established, and the paper calls the evaluation 'preliminary.'
  • domain assumption Baseline no-skill runs are representative of each agent's default ability on these tasks
    §5.1 attributes the deltas to adding Tactile; this presumes the baselines were run under matched conditions (same model version, prompts, step budget) and were not under-tuned. The paper does not specify the baseline configuration.
invented entities (1)
  • action-grounded interface state (candidate targets with source labels, roles, state, geometry, affordances, verification cues) independent evidence
    purpose: Model-facing abstraction replacing anonymous screenshots/coordinates so agents can ground, act, and verify against semantic objects
    A software abstraction, not a physical entity. Independent evidence is the open-source MCP repository and benchmark protocol, which others can run and falsify; internal evidence is the paper's own macOSWorld-style deltas.

pith-pipeline@v1.3.0-alltime-deepseek · 9030 in / 16170 out tokens · 159246 ms · 2026-08-02T02:04:07.004881+00:00 · methodology

0 comments
read the original abstract

Computer-use agents are becoming capable software operators, but their interface to desktop applications is still often a brittle motor layer: they look at screenshots, predict coordinates, click, and hope that the visible state changed as intended. This collapses target grounding, action execution, and outcome verification into a single ambiguous operation. We present Tactile, an open-source tool layer that gives agents a more reliable "hands and feet" for desktop use. Tactile converts heterogeneous UI evidence--operating-system accessibility semantics, OCR-grounded text, and visual fallback regions--into action-grounded interface states: compact target candidates with source labels, roles or text, state, geometry, executable affordances, and verification cues. Agents operate through an observe-ground-act-verify loop that prefers native semantic actions when available, falls back to OCR-grounded coordinates when visible text is the best evidence, and keeps full provenance for replay and failure attribution. On macOSWorld-style tasks, adding Tactile improves Codex Success@100 from 41.1% to 50.0% overall and from 45.2% to 55.3% on accessibility-adapted tasks; a 96-task cross-agent subset shows consistent gains across Codex, Claude Code, OpenCode, and Goose. These results suggest that reliable computer use requires not only stronger models, but also a reusable execution substrate that exposes software actions as semantic, verifiable, and auditable objects rather than anonymous screen coordinates.

Figures

Figures reproduced from arXiv: 2607.14443 by Yong Liu, Zhanpeng Shi, Zhenyi Zhong.

Figure 1
Figure 1. Figure 1: Success rate versus average task runtime for four agents, with and without [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Motivation and overview. Screenshot-first control is general but often entangles perception, target selection, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: TACTILE system architecture. Agent clients call a standardized MCP tool surface. The TACTILE server compiles raw accessibility, OCR, and visual evidence into compact observations, executes actions through a local macOS runtime, and returns verification evidence while preserving full traces for replay and debugging. 4 System Design and Implementation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: summarizes the benchmark results. In the graded Codex comparison, adding TACTILE raises Success@100 from 41.06% to 50.00% overall, an 8.94 percentage-point gain. The gain is larger on AX-adapted tasks, rising from 45.22% to 55.26% (+10.04 points). On Limited-AX tasks, success improves from 27.78% to 33.33% (+5.55 points). This pattern is consistent with the operating ladder: accessibility semantics provide… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 12 linked inside Pith

  1. [1]

    Mind2Web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web, 2023. URLhttps://arxiv.org/abs/2306.06070

    Pith/arXiv arXiv 2023

  2. [2]

    GPT-4V(ision) is a generalist web agent, if grounded, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded, 2024. URLhttps://arxiv.org/abs/2401.01614

    Pith/arXiv arXiv 2024

  3. [3]

    WebV oyager: Building an end-to-end web agent with large multimodal models, 2024

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebV oyager: Building an end-to-end web agent with large multimodal models, 2024. URL https: //arxiv.org/abs/2401.13919

    Pith/arXiv arXiv 2024

  4. [4]

    ScreenAgent: A vision language model-driven computer control agent, 2024

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. ScreenAgent: A vision language model-driven computer control agent, 2024. URL https://arxiv.org/abs/ 2402.07945

    Pith/arXiv arXiv 2024

  5. [5]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URLhttps://arxiv.org/abs/2404.07972

    Pith/arXiv arXiv 2024

  6. [6]

    AndroidWorld: A dynamic benchmarking environment for autonomous agents, 2024

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A dynamic benchmarking environment for autonomous agents, 2024. URLhttps://arxiv.org/abs/2405.14573

    Pith/arXiv arXiv 2024

  7. [7]

    macOSWorld: A multilingual interactive benchmark for gui agents,

    Pei Yang, Hai Ci, and Mike Zheng Shou. macOSWorld: A multilingual interactive benchmark for gui agents,

  8. [8]

    SeeClick: Harnessing gui grounding for advanced visual gui agents, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing gui grounding for advanced visual gui agents, 2024. URL https://arxiv.org/abs/2401.10935

    Pith/arXiv arXiv 2024

  9. [9]

    OmniParser for pure vision based gui agent,

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. OmniParser for pure vision based gui agent,

  10. [10]

    Accessibility programming guide for OS X

    Apple Developer Documentation. Accessibility programming guide for OS X. https://developer.apple. com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/ , 2015. Apple Documentation Archive; accessed 2026-06-29

    2015

  11. [11]

    The OS X accessibility model

    Apple Developer Documentation. The OS X accessibility model. https://developer.apple.com/ library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXmodel. html, 2015. Apple Documentation Archive; accessed 2026-06-29

    2015

  12. [12]

    AXUIElement.h

    Apple Developer Documentation. AXUIElement.h. https://developer.apple.com/documentation/ applicationservices/axuielement_h, 2026. Apple Developer Documentation; accessed 2026-06-29

    2026

  13. [13]

    AXUIElementPerformAction(_:_:)

    Apple Developer Documentation. AXUIElementPerformAction(_:_:). https://developer.apple.com/ documentation/applicationservices/1462091-axuielementperformaction, 2026. Apple Developer Documentation; accessed 2026-06-29

    arXiv 2026

  14. [14]

    UI Automation Overview

    Microsoft Learn. UI Automation Overview. https://learn.microsoft.com/en-us/windows/win32/ winauto/uiauto-uiautomationoverview, 2025. Win32 apps documentation, updated 2025-07-14; accessed 2026-06-29

    2025

  15. [15]

    UI Automation Control Patterns Overview

    Microsoft Learn. UI Automation Control Patterns Overview. https://learn.microsoft.com/en-us/ windows/win32/winauto/uiauto-controlpatternsoverview, 2025. Win32 apps documentation; accessed 2026-06-29

    2025

  16. [16]

    Web content accessibility guidelines (WCAG) 2.2

    World Wide Web Consortium. Web content accessibility guidelines (WCAG) 2.2. https://www.w3.org/TR/ WCAG22/, 2024. W3C Recommendation, 12 December 2024; accessed 2026-06-29

    2024

  17. [17]

    Accessible rich internet applications (W AI-ARIA) 1.2.https://www.w3.org/ TR/wai-aria-1.2/, 2023

    World Wide Web Consortium. Accessible rich internet applications (W AI-ARIA) 1.2.https://www.w3.org/ TR/wai-aria-1.2/, 2023. W3C Recommendation, 6 June 2023; accessed 2026-06-29

    2023

  18. [18]

    Screen2AX: Vision-based approach for automatic macos accessibility generation, 2025

    Viktor Muryn, Marta Sumyk, Mariya Hirna, Sofiya Garkot, and Maksym Shamrai. Screen2AX: Vision-based approach for automatic macos accessibility generation, 2025. URLhttps://arxiv.org/abs/2507.16704

    Pith/arXiv arXiv 2025

  19. [19]

    Model context protocol specification

    Model Context Protocol. Model context protocol specification. https://modelcontextprotocol.io/ specification/2025-11-25, 2025. Version 2025-11-25; accessed 2026-06-29

    2025

  20. [20]

    Model context protocol: Server tools specification

    Model Context Protocol. Model context protocol: Server tools specification. https://modelcontextprotocol. io/specification/2025-11-25/server/tools, 2025. Version 2025-11-25; accessed 2026-06-29. 10 TACTILE

    2025

  21. [21]

    ReAct: Synergizing reasoning and acting in language models, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2022. URLhttps://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2022

  22. [22]

    UI-TARS: Pioneering automated GUI interaction with native agents, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Jiang Xu, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Li...

    Pith/arXiv arXiv 2025

  23. [23]

    CUA-Skill: Develop skills for computer using agent, 2026

    Tianyi Chen, Yinheng Li, Michael Solodko, Sen Wang, Nan Jiang, Tingyuan Cui, Junheng Hao, Jongwoo Ko, Sara Abdali, Qing Xiao, Leon Xu, Suzhen Zheng, Hao Fan, Pashmina Cameron, Justin Wagle, and Kazuhito Koishida. CUA-Skill: Develop skills for computer using agent, 2026. URLhttps://arxiv.org/abs/2601.21123

    arXiv 2026

  24. [24]

    ScreenCaptureKit

    Apple Developer Documentation. ScreenCaptureKit. https://developer.apple.com/documentation/ screencapturekit, 2026. Apple Developer Documentation; accessed 2026-06-29

    2026

  25. [25]

    RecognizeTextRequest.https://developer.apple.com/documentation/ vision/recognizetextrequest, 2026

    Apple Developer Documentation. RecognizeTextRequest.https://developer.apple.com/documentation/ vision/recognizetextrequest, 2026. Apple Developer Documentation; accessed 2026-06-29

    2026

  26. [26]

    Quartz Event Services

    Apple Developer Documentation. Quartz Event Services. https://developer.apple.com/documentation/ coregraphics/quartz-event-services, 2026. Apple Developer Documentation; accessed 2026-06-29

    2026

  27. [27]

    Codex CLI

    OpenAI. Codex CLI. https://developers.openai.com/codex/cli, 2026. OpenAI developer documenta- tion; accessed 2026-06-29

    2026

  28. [28]

    Claude Code

    Anthropic. Claude Code. https://claude.com/product/claude-code , 2026. Product documentation; accessed 2026-06-29

    2026

  29. [29]

    OpenCode: The open source ai coding agent

    OpenCode Project. OpenCode: The open source ai coding agent. https://opencode.ai/ , 2026. Project documentation; accessed 2026-06-29

    2026

  30. [30]

    goose: Your open source ai agent

    goose Project. goose: Your open source ai agent. https://goose-docs.ai/, 2026. Project documentation; accessed 2026-06-29

    2026

  31. [31]

    Rethinking irregular scene text recognition, 2019

    Shangbang Long, Yushuo Guan, Bingxuan Wang, Kaigui Bian, and Cong Yao. Rethinking irregular scene text recognition, 2019. URLhttps://arxiv.org/abs/1908.11834

    Pith/arXiv arXiv 2019

  32. [32]

    mcp-server-macos-use

    mediar-ai. mcp-server-macos-use. https://github.com/mediar-ai/mcp-server-macos-use , 2026. GitHub repository; accessed 2026-06-29. 11

    2026

  33. [2024]

    URLhttps://arxiv.org/abs/2408.00203

    Pith/arXiv arXiv

  34. [2025]

    URLhttps://arxiv.org/abs/2506.04135

    arXiv