Pith. sign in

REVIEW 4 major objections 5 minor 17 references

StateAct makes program state the primary interface for computer-use agents, lifting a frontier model's binary success on OSWorld 2.0 from 20.6% to 26.9% at about nine times lower cost, and argues the bottleneck shifts from perception to rea

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:02 UTC pith:JF2EX2RF

load-bearing objection StateAct is a well-engineered systems paper with an honest failure analysis, but the headline +6.3 binary gain rests on single runs and is within plausible seed noise. the 4 major comments →

arxiv 2607.22798 v1 pith:JF2EX2RF submitted 2026-07-24 cs.SE cs.CV

StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

Yan Yang , Xiangru Jian , Ziyang Luo , Zirui Zhao , Yutong Dai , Ziji Shi , Hanshu Yan , Jun Hao Liew
show 2 more authors
Silvio Savarese Junnan Li
This is my paper
classification cs.SE cs.CV
keywords computer-use agentsprogram statestate groundingGUI agentslong-horizon tasksverificationmulti-agent harnessOSWorld 2.0
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that a computer-use agent should act on program state—files, application backends, and DOM—rather than on screenshots, because a screenshot is a lossy rendering that hides the values and formulas that determine whether a desktop task succeeded. It introduces StateAct, a multi-agent harness whose main agent reads and writes state through code, delegates the few irreducibly visual subgoals to a GUI specialist, and verifies completion with an independent gate that inspects the persisted artifact. On OSWorld 2.0 this harness alone lifts Claude Opus 4.8 from 20.6% to 26.9% binary success and from 54.8% to 61.6% partial success at about nine times lower cost. The paper argues that grounding action, verification, and memory in state moves the accuracy wall from perception to reasoning: failures now depend on what the agent thinks, not what it sees.

Core claim

StateAct is built on the observation that a screenshot is a non-injective render map: many distinct program states can produce the same pixels, so no perceptual model can in general recover the state that matters. The paper defines a state channel that queries files, backends, or DOM directly, claims it is effectively invertible over the substate a task touches, and constructs an agent loop in which a main agent acts through code on that channel, a GUI subagent handles screen-based interaction for the rare subgoal (1.1% of steps, 28 of 108 tasks), and a narration-blind finish gate re-reads the persisted deliverable to catch structural failures. The result is that the same backbone model impr

What carries the argument

The central object is the state channel, a query map that reads and writes program state directly via bash, Python, file editors, and DOM serialization, as opposed to the pixel channel that renders the screen. Its load-bearing property is effective invertibility over the substate a task touches, which lets the main agent read and write exact values (e.g., a spreadsheet formula) instead of ambiguous renderings. Around it the paper builds three mechanisms: state-grounded action, a narration-blind finish gate that independently re-reads the persisted artifact, and context management—fresh-context subagent delegation, compaction, and an externalized plan checklist—that sustains the agent over hu

Load-bearing premise

The claim rests on the assumption that, for the parts of a task that matter, the program state can be reliably read and written through code—that the state channel is effectively invertible—so tasks whose deliverable lives only in rendered, encrypted, or undocumented form would give the code-first agent no advantage.

What would settle it

Measure the fraction of tasks in a long-horizon suite where the agent's state-probing commands (bash/Python/DOM queries) fail to locate or write the target value before the GUI subagent is invoked. If a code-first agent shows no gain over a screenshot baseline on tasks whose state is invisible to code, the invertibility premise is disproved. Alternatively, benchmark StateAct on a render-only suite (image editing, canvas layout) and check whether its advantage disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On the tested benchmark, grounding the main agent in program state rather than screenshots improves the same backbone model's binary success from 20.6% to 26.9% and partial success from 54.8% to 61.6%, while cutting per-task cost about ninefold.
  • The benefit comes from what the agent observes, not from added agentic depth: an ablation shows removing act-on-state causes the largest drop (from 61.6% to 51.3% partial), and the recursion mechanism fires on only 7 of 108 tasks.
  • A code-only variant without the GUI subagent reaches only 45.9% partial, below the 54.8% screenshot baseline, so some visual subgoals still require screen interaction.
  • On a short-horizon benchmark the harness and the reference perform almost identically (78.4% vs 77.3% binary), indicating the advantage is specific to long-horizon tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the state channel's effective invertibility is the load-bearing property, the design's reach is bounded by how often applications expose their state to code; a testable extension would instrument the harness to measure state-probe failure rates per task, which the paper does not report.
  • The paper's own taxonomy predicts that render-only subgoals (image editing, layout, WYSIWYG) will remain outside the state channel's benefit; a direct comparison on a render-only benchmark would test this boundary.
  • The verifier's 68/76 false-pass rate suggests that structural checks alone are insufficient; an obvious next step is a verifier that re-derives values from multiple independent sources instead of the same source under the same interpretation.
  • Since a compact GUI specialist nearly matches a frontier one on four of five benchmarks, the authors' implicit suggestion is that investing in the main agent's state-handling ability may be more cost-effective than scaling the vision model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes StateAct, a code-first multi-agent harness for long-horizon computer-use agents. The main agent acts on program state through bash, Python, and file/DOM reads rather than screenshots; a dedicated GUI subagent is invoked only when a subgoal is irreducibly visual. An independent, narration-blind finish gate checks the persisted artifact, and a context manager with compaction, fresh-context delegation, and an externalized plan sustains long episodes. On OSWorld 2.0 with Claude Opus 4.8 fixed as the backbone, StateAct is reported to raise binary success from 20.6% to 26.9% and mean partial success from 54.8% to 61.6%, at about 9x lower cost than the reference CUA harness. Ablations attribute most of the gain to act-on-state; the finish gate catches structural defects but wrongly passes 68 of 76 non-perfect tasks; a manual audit attributes 38 of 79 non-perfect tasks to reasoning errors. The paper concludes that state-grounding moves the accuracy wall from perception to reasoning, not removing it.

Significance. If the headline comparison is reliable, state-grounding is an important and cost-effective direction: it improves quality and cost simultaneously on the same backbone, with a clear architectural story and a falsifiable prediction about state-addressable versus render-only tasks. The paper is unusually honest about its verifier ceiling (68/76 wrong passes), the flat-delegation finding (recursion fires on 7/108 tasks and never nests), and the boundary where state-grounding should not help. The cost reduction is measured in tokens and is less sensitive to stochastic variation. However, the significance is conditional on the single-run empirical gain and on the external reference baseline; the diagnosis of a reasoning bottleneck is based on a manual audit of those same single trajectories.

major comments (4)
  1. [§5.2, Table 1] The central empirical claim is a single run per condition: StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% binary and from 54.8% to 61.6% partial on 108 tasks. The binary gain is about 7 task-level flips (29 vs 22 successes), and the harness is stochastic at multiple points (LLM sampling, delegation, compaction, gate retries). No variance, confidence interval, or paired significance test is reported. Because Table 3, Figure 7, and Figure 8 are all derived from the same trajectories, they cannot independently corroborate the gain. Please report multiple seeds for the main result and key ablations, and/or a per-task paired analysis (e.g., McNemar on the 2x2 success table, or a paired bootstrap on partial scores) with the seed spread.
  2. [§5.1, Table 1] The same-backbone reference baseline (Opus 4.8 CUA, 20.6/54.8, about $72/task) is taken from Yuan et al. (2026), not reproduced in-house. The claim of 'replacing the harness' requires matching model version, environment, screenshot resolution, sampling parameters, and turn budget. Cross-paper differences in any of these could be worth several points. Please reproduce the reference in the same harness and API configuration, or at minimum report its exact configuration and run a matched subset comparison. The ~9x cost reduction inherits this issue because the numerator and denominator come from different setups.
  3. [§6.1, Table 5 and Figure 8] The conclusion that 'the accuracy wall moves from perception to reasoning' rests on the manual audit in Figure 8: 38 of 79 non-perfect tasks are labeled reasoning errors. No annotation rubric, inter-annotator agreement, or blind adjudication is provided, and the labels were assigned after inspecting trajectories from the same model whose behavior is being classified. The finish gate's 68/76 wrong-pass rate is reported honestly, but the perception-versus-reasoning separation is not independently established. Please provide the annotation protocol, agreement rates, and ideally a subset check with an oracle or model-free verifier.
  4. [§3] The core premise is that 'over the substate a task touches, the state channel is effectively invertible' and that only 28/108 tasks require the GUI subagent. This is argued conceptually but never quantified. The paper does not report, per task, whether a state path was found, how many probing attempts failed before falling back to the GUI, or how the state-addressable/hybrid/render-only trichotomy was assigned. Without this, the claimed boundary and the generality of the principle are untested. Please add a per-task state-addressability analysis and show that the StateAct-vs-reference margin correlates with it.
minor comments (5)
  1. [§7] Typo: 'makes program state as the primary interface' should read 'makes program state the primary interface'.
  2. [§5.1] No statement of the number of seeds, sampling temperature, or random seed is given. A short reproducibility paragraph would be valuable.
  3. [Table 2] The reference row reports only overall turns because per-capability CUA trajectories are unavailable. Please make the reason explicit in the caption to avoid implying a per-capability cost comparison.
  4. [Figure 8] The figure is dense and the leaf labels are hard to read; a tabular breakdown of the manual audit counts would be more accessible.
  5. [§6.2, Table 6] Since the OSWorld 2.0 headline is binary success, report binary as well as mean-partial for the five-benchmark subagent swap, not only mean-partial.

Circularity Check

0 steps flagged

No significant circularity: the central empirical claims are benchmarked against external OSWorld 2.0 labels and an independently reported baseline, with no parameter fitted to the target result.

full rationale

The paper's central claim—that StateAct raises Claude Opus 4.8 from 20.6% to 26.9% binary and 54.8% to 61.6% partial success on OSWorld 2.0—is evaluated against an external benchmark with success labels defined independently of StateAct, and the reference baseline is taken from the OSWorld 2.0 benchmark authors (Yuan et al., 2026), not from a parameter fitted by this paper. The state-grounding principle is argued conceptually (non-injective rendering, invertible state channel on the touched substate) rather than derived from the experimental result, so the experiments are tests, not tautologies. The finish gate does use the same model family as the main agent, but the paper explicitly identifies this as a common-mode limitation and measures its error rate (68/76 wrong passes) rather than concealing it; this is a limitation, not a definitional circularity. Self-citations appear only in related-work and baseline/ablation contexts (e.g., GTA1, CoAct-1, SFR-CUA) and are not load-bearing for the main result. The ablation and failure-taxonomy analyses are post-hoc interpretations of the same trajectories, but they are presented as diagnostics, not as independent confirmations. No equation or fitted parameter is shown to be equivalent to the predicted outcome by construction. Therefore no circular step meets the evidentiary bar required by the review rules.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No fitted statistical parameters; the free parameters are hand-chosen harness caps. The key unproven premise is state-channel invertibility, and the diagnosis depends on manual audit labels. No new physical entities are introduced.

free parameters (3)
  • Finish-gate retry cap = 3 rounds
    Chosen by hand (§4.2, §5.2); shapes the bounded-correction loop, with 26.7% of tasks retried and 21.4% of retries turning into binary success. Not fitted to data, but the measured gate behavior depends on it.
  • Subagent turn cap = 50
    Set in §5.1; limits each delegation loop and affects the ~155 model turns/task accounting.
  • Main-agent episode budget = 200 turns
    Episode cap in §5.1; any improvement in long-horizon execution is relative to this budget.
axioms (4)
  • domain assumption Over the substate a task touches, the state channel g is effectively invertible (Section 3, 'state channel is effectively invertible').
    The central premise of state-first action: code/state queries can read and write the artifact the task names. If an app stores state in an undocumented, encrypted, or visual-only form, this fails. Paper does not quantify how often OSWorld 2.0 tasks satisfy this.
  • domain assumption OSWorld 2.0 human-grader scores (binary and partial) are reliable ground truth.
    All comparisons in §5 use these labels; if partial-credit scoring is noisy, the +6.8 partial gain could be within labeling noise.
  • domain assumption The manual trajectory audit in §6.1 correctly classifies dominant failure causes.
    The 'reasoning 38' result that anchors the bottleneck-shift claim is based on manual audit labels, not on an independently executable metric.
  • domain assumption The rendering function is non-injective for the benchmark's task-critical states.
    Used in §3 to argue no perceptual model can recover state from pixels; intuitively true, but the paper does not measure state collisions on OSWorld 2.0.

pith-pipeline@v1.3.0-alltime-deepseek · 12346 in / 12453 out tokens · 113878 ms · 2026-08-01T05:02:38.114346+00:00 · methodology

0 comments
read the original abstract

Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.

Figures

Figures reproduced from arXiv: 2607.22798 by Hanshu Yan, Jun Hao Liew, Junnan Li, Silvio Savarese, Xiangru Jian, Yan Yang, Yutong Dai, Ziji Shi, Zirui Zhao, Ziyang Luo.

Figure 1
Figure 1. Figure 1: 26.9% binary success on Claude Opus 4.8: the only entry above the < 21% public frontier, at ∼ 9× lower cost than the same backbone’s reference computer-use-agent harness (∼ $7.8 vs. ∼ $72 per task). Orange marks the Claude Opus 4.8 reference baseline. StateAct ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: StateAct architecture. The main agent works directly with program state and uses fresh [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The same subgoal via each channel. The pixel channel yields a rendered, ambiguous value [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An independent finish gate transcript (condensed from the calendar-import failure class in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Three successful trajectories selected to illustrate state-first action, GUI delegation, and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cost–accuracy frontier on OSWorld 2.0 (binary success [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-capability performance across six systems on OSWorld 2.0’s ten multi-label capa [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-task dominant-cause audit of the 79 non-perfect tasks. Reasoning errors (38) domi￾nate; ∼ 20% hit modality bottlenecks or ambiguous instructions. Categories are manual audit labels from trajectory inspection [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 13 linked inside Pith

  1. [1]

    Let’s think in two steps: Mitigating agreement bias in MLLMs with self-grounded verification

    Moises Andrade, Joonhyuk Cha, Brandon Ho, Vriksha Srihari, Karmesh Yadav, and Zsolt Kira. Let’s think in two steps: Mitigating agreement bias in MLLMs with self-grounded verification. arXiv preprint arXiv:2507.11662,

    arXiv

  2. [5]

    ComputerRL: Scaling end-to-end online reinforcement learning for computer use agents.arXiv preprint arXiv:2508.14040,

    Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. ComputerRL: Scaling end-to-end online reinforcement learning for computer use agents.arXiv preprint arXiv:2508.14040,

    arXiv

  3. [6]

    Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651,

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651,

    Pith/arXiv arXiv

  4. [8]

    Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366,

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366,

    Pith/arXiv arXiv

  5. [9]

    CoAct-1: Computer-using multi-agent system with coding actions.arXiv preprint arXiv:2508.03923,

    Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, and Caiming Xiong. CoAct-1: Computer-using multi-agent system with coding actions.arXiv preprint arXiv:2508.03923,

    arXiv

  6. [10]

    Exe- cutable code actions elicit better LLM agents.arXiv preprint arXiv:2402.01030,

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better LLM agents.arXiv preprint arXiv:2402.01030,

    Pith/arXiv arXiv

  7. [11]

    Open- Computer: Verifiable software worlds for computer-use agents.arXiv preprint arXiv:2605.19769,

    Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou, Kangqi Ni, Guo Gan, and Arman Cohan. Open- Computer: Verifiable software worlds for computer-use agents.arXiv preprint arXiv:2605.19769,

    Pith/arXiv arXiv

  8. [12]

    Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454,

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454,

    Pith/arXiv arXiv

  9. [13]

    MCPWorld: A unified benchmarking testbed for API, GUI, and hybrid computer use agents

    Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, and Mengwei Xu. MCPWorld: A unified benchmarking testbed for API, GUI, and hybrid computer use agents. arXiv preprint arXiv:2506.07672,

    Pith/arXiv arXiv

  10. [14]

    Set- of-mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441,

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in GPT-4V.arXiv preprint arXiv:2310.11441,

    Pith/arXiv arXiv

  11. [15]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793,

    Pith/arXiv arXiv

  12. [16]

    GTA1: Gui test-time scaling agent.arXiv preprint arXiv:2507.05791, 2025a

    Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Silvio Savarese, Caiming Xiong, and Junnan Li. GTA1: Gui test-time scaling agent.arXiv preprint arXiv:2507.05791, 2025a. Yuhao Yang, Zhen Yang, Zi-Yi Dou, Anh Nguyen, Keen You, Omar Attia, Andrew Szot, Michael Feng, ...

    Pith/arXiv arXiv

  13. [17]

    UFO2: The desktop AgentOS.arXiv preprint arXiv:2504.14603,

    12 Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. UFO2: The desktop AgentOS.arXiv preprint arXiv:2504.14603,

    Pith/arXiv arXiv

  14. [2023]

    UI-TARS: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326,

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jia- hao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua ...

    Pith/arXiv arXiv

  15. [2024]

    Agentic reward modeling: Verifying GUI agent via online proactive interaction.arXiv preprint arXiv:2602.00575,

    Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, and Zhixiong Zeng. Agentic reward modeling: Verifying GUI agent via online proactive interaction.arXiv preprint arXiv:2602.00575,

    Pith/arXiv arXiv

  16. [2025]

    Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264,

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264,

    Pith/arXiv arXiv

  17. [2026]

    Mobile- World: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments.arXiv preprint arXiv:2512.19432,

    Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, and Yue Wang. Mobile- World: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments.arXiv preprint arXiv:2512.19432,

    arXiv