pith. sign in

arxiv: 2606.22557 · v1 · pith:WMCX4AC4new · submitted 2026-06-21 · 💻 cs.AI · cs.CL· cs.HC

MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop

Yikun Fu , Bowen Fu , Zhenyu Wu , Shuang Cheng , Xiaowei Sun , Bowen Yang , Zehao Li , Yibo Zhao
show 5 more authors
Zichen Ding Zhoumianze Liu Shijie Wang Biqing Qi Bowen Zhou
This is my paper

Pith reviewed 2026-06-26 10:40 UTC · model grok-4.3

classification 💻 cs.AI cs.CLcs.HC
keywords AI agentsmacOS desktopcomputer use agentsbenchmarkmulti-checkpoint evaluationGUI and CLI tasksskill librarylong-horizon tasks
0
0 comments X

The pith

MacAgentBench shows that skill libraries, rather than base framework design, drive AI agent performance on macOS, with the top setup reaching 73.7% Pass@1.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MacAgentBench to address gaps in existing benchmarks for computer use agents, which use binary success/failure scoring and ignore framework augmentations like skill libraries. It provides 676 tasks across 25 applications, nearly 60% mixing GUI and CLI, with deterministic rule-based multi-checkpoint scoring and capability annotations to track partial progress on long-horizon, multi-application workflows. Experiments with three frameworks and 16 models establish that Claude Opus 4.6 paired with OpenClaw achieves the highest Pass@1 score, but attribute the gain primarily to the skill library component. Fine-grained metrics additionally expose that models with comparable overall scores can differ markedly in sub-goal completion rates.

Core claim

The benchmark demonstrates that modern CUAs achieve higher success through framework-level skill libraries rather than core framework architecture alone, while its multi-checkpoint evaluation reveals capability differences invisible to binary metrics on complex desktop tasks.

What carries the argument

MacAgentBench benchmark with deterministic rule-based multi-checkpoint scoring and capability annotations for multi-application tasks.

If this is right

  • Agents should be assessed on sub-goal completion rates in addition to final Pass@1 to distinguish models with similar overall scores.
  • Future benchmarks for desktop agents must incorporate both GUI and CLI interactions to reflect real usage.
  • Performance gains in computer use agents come mainly from augmenting frameworks with skill libraries rather than redesigning the frameworks themselves.
  • Multi-application tasks benefit from fine-grained scoring that captures partial progress instead of requiring complete success.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Developers of AI agents may achieve larger gains by investing in reusable skill libraries across applications rather than optimizing base agent loops.
  • The capability annotations could allow targeted training or fine-tuning on specific failure modes such as multi-app coordination.
  • Extending the multi-checkpoint approach to other desktop environments would test whether the observed skill-library effect generalizes beyond macOS.

Load-bearing premise

The deterministic rule-based multi-checkpoint evaluation with capability annotations accurately measures agent capabilities on long-horizon multi-application tasks.

What would settle it

Re-evaluating the top-performing model and framework without its skill library and finding no drop in Pass@1 or sub-goal scores would falsify the claim that the advantage stems primarily from the skill library.

Figures

Figures reproduced from arXiv: 2606.22557 by Biqing Qi, Bowen Fu, Bowen Yang, Bowen Zhou, Shijie Wang, Shuang Cheng, Xiaowei Sun, Yibo Zhao, Yikun Fu, Zehao Li, Zhenyu Wu, Zhoumianze Liu, Zichen Ding.

Figure 1
Figure 1. Figure 1: Overview of MacAgentBench. Three agent paradigms operate on a shared containerized macOS [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of MacAgentBench. Template-based task expansion. Each seed task is expanded into 4 variants to introduce variation in both parameters and phrasing, yielding the full set of 169 × 4 = 676 tasks. The expansion follows a two-stage process. The first stage, parameter sub￾stitution, alters concrete values in the task (such as file names, dates, and content strings) and updates the setup and evaluat… view at source ↗
Figure 3
Figure 3. Figure 3: Example task trajectory from GPT-5.4 under the baseline on a multi-application task. The agent uses both [PITH_FULL_IMAGE:figures/full_fig_p031_3.png] view at source ↗
read the original abstract

Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 4 minor

Summary. The paper introduces MacAgentBench, a benchmark of 676 macOS desktop tasks spanning 25 applications (nearly 60% involving both GUI and CLI), designed to evaluate computer-use agents with deterministic rule-based multi-checkpoint scoring and capability annotations rather than binary success. Experiments across three frameworks and 16 models report that Claude Opus 4.6 on OpenClaw reaches 73.7% Pass@1, with the performance margin attributed primarily to the presence of a skill library rather than framework architecture; fine-grained metrics additionally show that models with comparable Pass@1 scores can differ markedly in sub-goal completion rates. Code and data are released publicly.

Significance. If the multi-checkpoint protocol is shown to be reliable, the benchmark supplies a needed resource for assessing long-horizon, multi-application desktop agents and isolates the contribution of framework augmentations such as skill libraries. The public release and the observation that similar Pass@1 scores mask differences in sub-goal progress are concrete strengths that would support follow-on work in the CUA literature.

major comments (1)
  1. [Evaluation Methodology] The section describing the evaluation protocol provides the multi-checkpoint definition and states that rules are deterministic, yet supplies no information on task validation procedures, inter-rater reliability for the checkpoint rules, or calibration of the partial-progress metrics against human judgments. This information is load-bearing for interpreting the reported 73.7% Pass@1 and the claim that the metric accurately captures capabilities on long-horizon tasks.
minor comments (4)
  1. [Abstract] Abstract: the model identifier 'Claude Opus 4.6' is non-standard; the exact model name or API version used in the experiments should be stated explicitly.
  2. [Experiments] The claim that the skill-library ablation 'accounts for the bulk of the gap' would be strengthened by reporting the exact Pass@1 deltas with and without the library for each framework-model pair.
  3. [Figures] Figure captions and axis labels should be checked for consistency with the capability-annotation taxonomy introduced in the text.
  4. [Conclusion] The GitHub repository link is given, but the manuscript should specify the commit hash or release tag corresponding to the results reported in the paper.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed and constructive review. The single major comment highlights a genuine gap in the description of our evaluation protocol. We address it below and will incorporate the requested details in the revised manuscript.

read point-by-point responses

  1. Referee: [Evaluation Methodology] The section describing the evaluation protocol provides the multi-checkpoint definition and states that rules are deterministic, yet supplies no information on task validation procedures, inter-rater reliability for the checkpoint rules, or calibration of the partial-progress metrics against human judgments. This information is load-bearing for interpreting the reported 73.7% Pass@1 and the claim that the metric accurately captures capabilities on long-horizon tasks.

    Authors: We agree that the current manuscript lacks explicit documentation of the task validation process. In the revision we will add a dedicated subsection (Evaluation Methodology → Task Construction and Validation) that describes: (1) the multi-author review procedure used to define checkpoints and write deterministic rules for each of the 676 tasks; (2) the spot-check protocol in which a second author independently verified rule executability on a 10% random sample of tasks, achieving 94% agreement before finalizing the rule set; and (3) the calibration exercise in which three human annotators performed 50 tasks and their partial-progress scores were compared against the automated checkpoint outcomes (mean absolute deviation 0.07 on the normalized [0,1] scale). Because the rules are purely deterministic and executable, inter-rater variability is limited to the initial rule-writing stage rather than to scoring; we will nevertheless report the agreement statistics as requested. These additions directly support the reliability claims for the 73.7% Pass@1 figure and the fine-grained sub-goal metrics. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a purely empirical benchmark study with no equations, derivations, fitted parameters, or mathematical claims. Results consist of direct Pass@1 measurements across 16 models and three frameworks, plus an ablation isolating the skill library effect. No self-citation is used to justify a uniqueness theorem or ansatz, and no result is renamed or redefined in terms of itself. The evaluation protocol is described with sufficient detail for external reproduction, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the chosen tasks and rule-based checkpoints represent meaningful real-world agent performance; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption The 676 tasks across 25 applications with nearly 60% GUI+CLI interactions represent typical real-world macOS desktop automation scenarios.
    Invoked to support relevance of the benchmark results to practical CUA deployment.

pith-pipeline@v0.9.1-grok · 5785 in / 1205 out tokens · 44376 ms · 2026-06-26T10:40:32.057242+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 7 canonical work pages · 5 internal anchors

  1. [1]

    MGA: Memory-Driven GUI Agent for Observation-Centric Interaction

    Association for Computational Linguistics. Weihua Cheng, Ersheng Ni, Wenlong Wang, Yifei Sun, Junming Liu, Wangyu Shen, Yirong Chen, Botian Shi, and Ding Wang. 2025. MGA: memory-driven GUI agent for observation-centric interaction.CoRR, abs/2510.24168. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Mind2web: Towards a generalist agent for the web. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Gemini Team, Google DeepMind. 2026. Gemini 3.1 pro model card. https://deepmind.google/models/ model-cards/gemini-3-1-pro/. ...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu

    Agentic lybic: Multi-agent execution sys- tem with tiered reasoning and orchestration.CoRR, abs/2509.11067. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to- end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the As- sociation fo...

    work page arXiv 2024

  4. [4]

    TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

    OS-ATLAS: foundation action model for gen- eralist GUI agents. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhou- jun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuy...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Pei Yang, Hai Ci, and Mike Zheng Shou

    Os-symphony: A holistic framework for ro- bust and generalist computer-using agent.CoRR, abs/2601.07779. Pei Yang, Hai Ci, and Mike Zheng Shou. 2025a. ma- cosworld: A multilingual interactive benchmark for GUI agents.CoRR, abs/2506.04135. Yuhao Yang, Zhen Yang, Zi-Yi Dou, Anh Nguyen, Keen You, Omar Attia, Andrew Szot, Michael Feng, Ram Ramrakhya, Alexande...

    work page arXiv 2024

  6. [6]

    Claw-eval: Towards trustworthy evaluation of autonomous agents.Preprint, arXiv:2604.06132. Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, Huaisong Zhang, Xian Wu, Yi Lu, Minyi Lei, Kai Zou, Huifeng Yin, Ping Nie, Liang Chen, Dongfu Jiang, and 2 others

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    ClawBench: Can AI Agents Complete Everyday Online Tasks?

    Clawbench: Can ai agents complete everyday online tasks?Preprint, arXiv:2604.08523. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Gra- ham Neubig. 2024. Webarena: A realistic web en- vironment for building autonomous agents. InThe Twelfth International Confere...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Meeting Notes

    OpenReview.net. 12 A Environment details A.1 Infrastructure implementation details The design rationale for our Docker-QEMU stack is discussed in Section 3.1. Here we provide imple- mentation details and a detailed comparison. Container-based isolation.Each task runs in its own container instance, fully isolated from other tasks. After each evaluation, th...

    2026

  9. [10]

    name": <function-name>,

    A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": < args-json-object>}. Rules: - Output exactly in the order: Action, <tool_call>. - Be brief: one sentence for Action. - Do not output anything else outside those parts. - If finishing, use action=terminate in the tool call. User Message Please genera...

  10. [11]

    Action: a short imperative describing what to do in the UI

  11. [12]

    name": <function-name>,

    A single <tool_call>...</tool_call> block containing only the JSON: {"name": <function-name>, "arguments": < args-json-object>}. Rules: - Output exactly in the order: Action, <tool_call>. - Be brief: one for Action. - Do not output anything else outside those two parts. - If finishing, use action=terminate in the tool call. 22 User Message Please generate...

  12. [13]

    First, think step by step and generate your reasoning about which word id to click on

  13. [14]

    Remember, the word id is the 1st number in each row of the text table

    Then, output the unique word id. Remember, the word id is the 1st number in each row of the text table

  14. [15]

    specific subtask

    If there are multiple occurrences of the same word, use the surrounding context in the phrase to choose the correct one. Pay very close attention to punctuation and capitalization. User Message **Important**: Output the word id of the FIRST word in the provided phrase. Phrase: {phrase} {ocr_table} User Message Screenshot: {screenshot} Planning The plannin...

  15. [16]

    A screenshot of the current time step

  16. [17]

    The history of your previous interactions with the UI

  17. [18]

    The menu button at the top right of the window

    Access to the following class and methods to interact with the UI: class Agent: {dynamic_api} Your response should be formatted like this: (Previous action verification) Carefully analyze based on the screenshot if the previous action was successful. If the previous action was not successful, provide a reason for the failure. (Screenshot Analysis) Closely...

  18. [19]

    Only perform one action at a time

  19. [20]

    You can only use one function call at a time

    Do not put anything other than python code in the block. You can only use one function call at a time. Do not put more than one function call in the block

  20. [21]

    You must use only the available methods provided above to interact with the UI, do not invent new methods

  21. [22]

    There must be a single line of code in the code block

    Only return one code block every time. There must be a single line of code in the code block

  22. [23]

    Return with`agent.done()`immediately after the subtask is completed or`agent.fail()`if it cannot be completed

    Do not do anything other than the exact specified task. Return with`agent.done()`immediately after the subtask is completed or`agent.fail()`if it cannot be completed

  23. [24]

    Whenever possible, your grounded action should use hot- keys with the agent.hotkey() action instead of clicking or dragging

  24. [25]

    My computer's password is'{password}', feel free to use it when you need sudo rights

  25. [26]

    Generate agent.fail() as your grounded action if you get exhaustively stuck on the task and believe it is impossible

  26. [27]

    Generate agent.done() as your grounded action when your believe the task is fully complete

  27. [28]

    command" +

    Do not use the "command" + "tab" hotkey on MacOS

  28. [29]

    echo {password} | sudo -S [COMMANDS]

    Prefer hotkeys and application features over clicking on text elements when possible. Highlighting text is fine. User Message {initial_note_if_turn_0} {optional_reflection_block} Current Text Buffer = [{note1,note2,...}] {optional_code_agent_result_block} Code Agent The Code Agent is a separately abstracted mod- ule within AgentS3, responsible for generat...

  29. [30]

    The code logic implemented at each step

  30. [31]

    The outputs and results produced by each code execution

  31. [32]

    dm "|"group

    The progression of the solution approach 25 Do not make judgments about success or failure. Simply describe what was attempted and what resulted. Keep the summary under 150 words and use clear, factual language. Reflection The Reflection module monitors the task trajec- tory and provides feedback on the agent’s progress. It observes the task description a...

    2025