Pith. sign in

REVIEW 3 major objections 6 minor 205 references

A GUI agent trained on real phones, hybrid GUI+CLI actions, and long-horizon RL can match or beat frontier models on everyday device tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 14:01 UTC pith:A2U5EOM6

load-bearing objection Solid industrial systems report with real mobile infrastructure and hybrid GUI+CLI; the MobileWorld-Real headline is the softest number, not the whole story. the 3 major comments →

arxiv 2607.28227 v1 pith:A2U5EOM6 submitted 2026-07-30 cs.AI cs.CV

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Hanzhang Zhou , Panrong Tong , Xu Zhang , Quyu Kong , Chenglin Cai , Tianyu Xia , Gongjie Zhang , Jianan Zhang
show 8 more authors
Long Li Long Chen Lei Wang Gaole Dai Pengxiang Li Liangyu Chen Yue Wang Steven Hoi
This is my paper
classification cs.AI cs.CV
keywords GUI agentsreal-device mobilehybrid GUI+CLIonline reinforcement learningdata flywheelproactive agentscross-platform workflowsGUI grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that GUI agents will only become useful general executors when they leave simulated benchmarks and are built around real devices, cross-platform workflows, hybrid GUI-plus-command-line actions, long-horizon reliability, proactive service, and automated self-improvement. Qwen-UI-Agent is their answer: one foundation agent spanning mobile, desktop, web, and search, backed by a large physical-phone runtime, thousands of concurrent sandboxes, a unified action space that can batch GUI and CLI steps in one turn, an agent-driven data flywheel, online RL on trajectories past 100 steps, and a harness that starts tasks from notifications and carries state across phone and computer. On that stack it reports state-of-the-art mobile success—82.1% on MobileWorld, 92.2% on a new real-device suite, 97.5% on AndroidDaily—and competitive scores against leading closed models on computer use, browser use, and grounding, while keeping general reasoning and agentic skill. A sympathetic reader cares because the claim is not a small benchmark bump: it is that co-designing model, environment, data loop, training, and harness around real-world use is what closes the gap between demo agents and something people can actually run on their phones and laptops.

Core claim

Qwen-UI-Agent shows that a real-world-centric foundation GUI agent—trained with real-device mobile experience, a hybrid GUI+CLI batched action space, an AutoResearch-style data flywheel, scalable online RL, and a proactive cross-platform harness—can set state-of-the-art mobile-use performance and stay competitive with frontier models on computer, browser, and grounding tasks without collapsing into a narrow GUI-only model.

What carries the argument

The integrated system: real-device mobile runtime plus scalable sandboxes; unified GUI+CLI and batched actions; agent-driven data flywheel; action RL then verifier-guided online RL on long trajectories; and a harness for notification-triggered proactive service and cross-device stateful workflows.

Load-bearing premise

The headline real-phone wins rest on a majority-vote vision-model judge being accurate enough to separate true task success from model mistakes and live-environment failures across the full comparison.

What would settle it

Have independent human experts re-score the full MobileWorld-Real trajectories for Qwen-UI-Agent and the top closed baselines under the same protocol; if the 92.2% lead shrinks or vanishes once env errors and judge disagreements are settled by people, the central real-device claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Mobile GUI agents should train and evaluate on physical devices and live apps, not only resettable sandboxes, if real-world reliability is the goal.
  • Computer-use agents can cut long trajectories by interleaving CLI with GUI and emitting multi-step batches in one model turn.
  • Capability growth can be driven mainly by agents that synthesize tasks, environments, and verifiers and diagnose failures, with humans in an oversight role.
  • A thin harness above the policy can turn notifications into proactive multi-step help and keep one workflow alive across phone and desktop.
  • Preserving general and agentic skill during GUI post-training is feasible and useful for out-of-distribution real requests.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If real-device closed-loop training is the main mobile gap, open agents without physical fleets may keep winning sandboxes and losing on live phones until shared real-device eval becomes standard.
  • Batched hybrid actions imply latency and cost gains that matter as much as success rate for products; step-count and wall-clock should become first-class leaderboard metrics.
  • Proactive notification harnesses raise a sharper safety and consent problem than reactive agents: false or over-eager interventions could erode trust faster than missed assists.
  • The same flywheel-plus-online-RL pattern is a template for other embodied digital agents (IDE, CRM, OS settings) where sim-to-real and long-horizon credit assignment dominate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This technical report presents Qwen-UI-Agent, a multi-domain foundation GUI agent spanning mobile, computer-use, browser, and DeepSearch. The system co-designs (i) scalable sandboxes plus a real-device mobile runtime (>100 phones, >150 apps), (ii) a unified action space with GUI, CLI, API, ask_user, and batched multi-actions per turn, (iii) an AutoResearch-style agent-driven data flywheel for task/environment/verifier synthesis and failure-driven iteration, (iv) SFT plus Action RL and verifier-guided online RL (GRPO-style) with up to ~10k concurrent environments and trajectories >100 steps, and (v) a harness for notification-driven proactive service and cross-platform workflows. Empirically, the 27B primary model reports SOTA mobile results (82.1% MobileWorld GUI-only, 92.2% MobileWorld-Real, 97.5% AndroidDaily), competitive computer-use (79.5% OSWorld-Verified; 40.0% partial / 13.9% binary on OSWorld-v2), 73.6% WebArena, strong grounding (e.g., 81.5% ScreenSpot-Pro with zoom), and retention of general/agentic capability versus the Qwen base. Behavioral analyses document real-device failure modes, GUI–CLI/batch usage, and RL-induced verification and loop-breaking behaviors.

Significance. If the results hold under fair comparison, this is a substantial systems contribution to GUI agents: it treats real-device execution, hybrid GUI+CLI batching, long-horizon online RL infrastructure, automated data flywheels, and proactive/cross-platform harnesses as first-class co-designed components rather than isolated benchmark tricks. Strengths include breadth of external benchmarks (OSWorld, WebArena, ScreenSpot suite, BrowseComp), explicit behavioral analysis (§4) that goes beyond leaderboard numbers, and transparent limitations on AutoJudge and incomplete 35B CUA/DeepSearch training. The real-device runtime, curriculum online RL, and hybrid action design are practically valuable and likely to influence follow-on work even if individual SOTA margins move. The proactive harness is more qualitative but points at a user-facing direction beyond reactive task completion.

major comments (3)
  1. [§3.2, Table 3, Appendix A.1] §3.2–3.3.1, Table 3, Limitations, Appendix A.1: The headline real-device claim (92.2% MobileWorld-Real, leads of 3.5–7.5 pp over Seed 2.1 Pro / Gemini 3.1 Pro / GPT-5.6 Sol / Opus 4.8) rests on AutoJudge (5-VLM majority; pass/failed/env_error; env_error excluded from the denominator). Expert agreement is 92.8% overall on 666 trajectories, but only 85.7% on model failure and 76.0% on env_error—the boundary that changes both numerator and denominator. Same protocol on all systems does not rule out differential bias (exploration style, trajectory shape, live-app dead ends). Please report: (i) per-model env_error rates and sensitivity of rankings when env_error is counted as failure or when disputed cases are re-labeled by humans; (ii) stratified agreement on non-Qwen trajectories; (iii) bootstrap/CI on Table 3 gaps. Sandbox MobileWorld and AndroidDaily already support a strong mobile story;
  2. [§3.3.2–3.3.3, Tables 4–7] §3.3.2–3.3.3, Tables 4–7: Several strongest closed-model comparisons mix official numbers with author re-runs (WebArena baselines marked *; OSWorld-v2 action-mode differs single vs batched). For WebArena the authors corrected reference answers/scripts—appropriate, but success rates are then not strictly comparable to uncorrected public numbers. Please add a short protocol appendix: decoding/step budgets, tool access, whether opponents used CLI/batch/API on OSWorld, and side-by-side scores under identical harness. Without this, “competitive with Opus 4.8 / GPT-5.x” on computer/browser is harder to audit than the mobile sandbox result in Table 2.
  3. [§2.4, §3.4, §4] §2.4–2.5 and §3.4: Central systems claims (Action RL, online RL curriculum, hybrid GUI+CLI, harness) are supported mainly by end-to-end scores plus qualitative/behavioral evidence (§4.2–4.4, Tables 11–13). Table 12 and the verification/false-stop deltas are useful but narrow. Load-bearing ablations are missing or underspecified: SFT-only vs +Action RL vs +Online RL on MobileWorld / OSWorld-Verified; GUI-only vs GUI+CLI on OSWorld; with/without batching (steps and success); harness vs base agent on a small fixed proactive/cross-platform suite with success criteria. Even a compact ablation table would separate infrastructure contribution from scale and data volume.
minor comments (6)
  1. [§1] Introduction says “five transitions” after listing six key transitions (§1); align the count.
  2. [§2.4.2] Eq. (5) Action RL reward: define ranges/normalization of C_t, Q_t and default weights (w_type, w_arg, λ_sens, λ_rep); currently not fully specified for reproduction.
  3. [Abstract, Figure 1] Figure 1 / abstract model names (Opus 4.8, GPT-5.6 Sol, Gemini 3.1 Pro, etc.) should be consistently tied to citations and access dates in tables.
  4. [§3.3.5, Table 9] Table 9 notes extensive judge/harness substitutions (GPT-5.5, etc.); consider a one-column “delta vs base” summary to make retention vs GUI-specialists easier to read.
  5. [§2.5, §3.4] Proactive/cross-platform results are demo trajectories (§3.4, Figs. 11–12, 28–29). A small quantitative suite (even N≈20–50 scripted scenarios) would strengthen §2.5 claims without blocking the paper.
  6. [Abstract, §2.5, Table 6] Minor typos/consistency: “support training” → “supports training” (abstract); “to to actionable” (§2.5); clarify 35A3B vs 35B-A3B naming in Table 6.

Circularity Check

0 steps flagged

Empirical systems/ML report: no derivation chain; SOTA claims are measured outcomes on held-out and external benchmarks, not quantities forced by construction.

full rationale

Qwen-UI-Agent is a technical report whose load-bearing claims are empirical success rates (MobileWorld 82.1%, MobileWorld-Real 92.2%, AndroidDaily 97.5%, OSWorld-Verified 79.5%, WebArena 73.6%, ScreenSpot-Pro 81.5%, etc.) obtained after SFT, action RL, and online RL. There is no first-principles derivation in which a predicted quantity is algebraically or statistically identical to a fitted input. Training uses agent-synthesized tasks and verifiers and a GRPO-style objective with binary verifier rewards; evaluation is reported on external suites (OSWorld, WebArena, BrowseComp, ScreenSpot family, AndroidDaily, MobileWorld sandbox) plus author-introduced MobileWorld-Real with tasks held out from training. Continuation of MAI-UI and reuse of step-level VLM judges / GRPO variants are ordinary self-citation of prior methods, not uniqueness theorems or ansatz-smuggling that force the headline numbers. AutoJudge’s env_error exclusion and majority-vote protocol raise evaluation-validity risk (judge bias, class confusion on failed vs env_error), but that is not circularity: under a fixed protocol different models still receive different scores, and the metric is not defined to equal a training fit. No self-definitional loop, fitted-input-as-prediction, or renaming of a known law appears in the manuscript.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

Load-bearing premises are engineering and evaluation assumptions rather than mathematical axioms: that sandbox+real-device experience closes sim-to-real gap; that VLM judges and code verifiers supply faithful rewards; that GRPO-style group-relative advantages on binary verifiers improve long-horizon policy; that notification-derived 'affairs' are a sufficient proactive interface. Free parameters are training/design choices (window size, reward weights, concurrent env count, model sizes). Invented entities are system components (AutoJudge, harness affair abstraction, MobileWorld-Real), not physical postulates.

free parameters (5)
  • SFT sliding window n=5 with stride 4 = n=5, overlap=1
    Chosen packing hyperparameter for long-trajectory SFT efficiency; affects what visual context each supervised action sees.
  • Action RL reward weights (w_type, w_arg, λ_sens, λ_rep) and format gate F_t
    Hand-designed action-level reward terms that directly shape which local errors are suppressed.
  • Online RL group size K and active/monitoring curriculum thresholds
    Curriculum admits tasks by empirical success-rate bands; thresholds determine which tasks provide GRPO signal.
  • AutoJudge panel size and majority vote (5 VLMs) = 5 judges, majority vote
    Evaluation hyperparameter that defines MobileWorld-Real labels used for the headline real-device claim.
  • Model scale choices (27B primary, 35B-A3B, 4B) = 27B primary
    Primary reported agent is the 27B variant; capability claims depend on this capacity and base checkpoint.
axioms (5)
  • domain assumption Real-device interaction distributions are necessary and sufficient to close the mobile sim-to-real gap that sandboxes miss (pop-ups, permissions, CAPTCHAs, super-app UI).
    Stated throughout §1–§2.2.2 and used to justify the 100+ phone runtime and MobileWorld-Real.
  • domain assumption Step-level VLM-as-judge filtering yields SFT signal comparable or better than verifier-selected full trajectories; executable verifiers are reliable enough for online RL outcome rewards.
    §2.3 and §2.4.3; training quality hinges on judge/verifier faithfulness.
  • domain assumption Group-relative policy optimization on binary terminal rewards, with model-adaptive task curriculum, improves long-horizon GUI decision making without collapsing useful exploration.
    §2.4.3; central to claims about >100-step training.
  • ad hoc to paper Mobile notifications plus affair/event/task memory are an adequate and user-controllable interface for proactive service initiation.
    §2.5.1 design choice; qualitative demos support utility but no large controlled user study is reported.
  • domain assumption Standard multimodal agent assumptions: screenshots + history suffice for policy πθ; coordinates and bash side effects are faithfully executed by environment adapters.
    Task formulation §2.1; shared by the GUI-agent literature.
invented entities (4)
  • MobileWorld-Real benchmark no independent evidence
    purpose: Measure live Android Chinese-ecosystem task success beyond sandboxes.
    Author-introduced 409-task/104-app real-device suite; central to SOTA mobile claim.
  • AutoJudge independent evidence
    purpose: Trajectory-level pass/fail/env_error labeling when apps lack programmatic verifiers.
    VLM majority-vote judge validated at 92.8% on 666 expert-labeled trajectories; still author-controlled evaluation machinery.
  • Proactive harness 'affair' abstraction no independent evidence
    purpose: Persistent real-world matter linking notifications, memory, and executable tasks across time.
    New systems abstraction for proactive/cross-platform behavior; evidenced only by qualitative demos in-paper.
  • Unified hybrid GUI+CLI+API batched action space for Qwen-UI-Agent independent evidence
    purpose: Allow one policy turn to interleave clicks, bash, APIs, ask_user, and multi-action batches.
    Core interface design; usage stats on OSWorld support it empirically inside this paper.

pith-pipeline@v1.2.0-daily-grok45 · 54426 in / 4158 out tokens · 82884 ms · 2026-07-31T14:01:52.999548+00:00 · methodology

0 comments
read the original abstract

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.

Figures

Figures reproduced from arXiv: 2607.28227 by Chenglin Cai, Gaole Dai, Gongjie Zhang, Hanzhang Zhou, Jianan Zhang, Lei Wang, Liangyu Chen, Long Chen, Long Li, Panrong Tong, Pengxiang Li, Quyu Kong, Steven Hoi, Tianyu Xia, Xu Zhang, Yue Wang.

Figure 1
Figure 1. Figure 1: Qwen-UI-Agent demonstrates leading or competitive performance across diverse GUI settings. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustrative trajectory of Qwen-UI-Agent for proactive cross-platform task execution. The left [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The environment infrastructure of Qwen-UI-Agent. (a) Scalable sandbox environments span [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Real-device mobile runtime with closed-loop environment governance. The health-aware [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The data flywheel of Qwen-UI-Agent. Domain capability bootstrapping initializes training, [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of our harness for proactive service initiation and cross-platform execution. (I) [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Overview of MobileWorld-Real, a real-device benchmark with human-written tasks that reflect the breadth of everyday mobile use. Representative examples and aggregate statistics show broad domain coverage and a long-tailed App distribution. Matched-model results further show lower success rates and longer trajectories than on AndroidDaily, highlighting the challenge of real-world mobile interaction. MobileW… view at source ↗
Figure 8
Figure 8. Figure 8: Demonstration of real-device mobile GUI execution. The trajectory is rendered as key frames with the executed action annotated beneath each frame. In this cross-app task, the agent looks up the target address on Amap, finds the most popular café nearby on Dianping, and posts a summary of the findings on RedNote. planning, visual understanding, and reliable execution in complex mobile environments. We furth… view at source ↗
Figure 9
Figure 9. Figure 9: Demonstration of hybrid GUI+CLI execution in computer-use tasks. Key frames are shown with the emitted actions, where CLI commands are highlighted in blue, GUI actions in red, and each step carries a short annotation of its intent. In this example, the agent selects the target photo and filters by stitching all candidates into a single montage via CLI for one-shot visual inspection, rather than opening eac… view at source ↗
Figure 10
Figure 10. Figure 10: Demonstration of DeepSearch-assisted GUI execution. The upper panels show the multi￾round search process, including keyword-based DeepSearch, targeted web fetches with intermediate thoughts, and the final search summary; the lower row shows the subsequent GUI trajectory. DeepSearch resolves the knowledge and reasoning problem before GUI execution, converting the vague cross-source request into an explicit… view at source ↗
Figure 11
Figure 11. Figure 11: Demonstration of proactive service based on mobile notifications. The trajectory is orga￾nized into highlighted stages: proactive task initiation from a detected notification, proactive execution phases, and a decision-ready result, with the executed action annotated beneath each key frame. Flight￾cancellation recovery: when the user’s next-morning flight is canceled and a 14:00 demo is at risk, the harne… view at source ↗
Figure 12
Figure 12. Figure 12: Demonstration of cross-platform task execution. In this workflow, mobile subtasks run on virtual screens of the physical device, so execution does not block the user’s own actions. Parallel multi-app search: the agent searches sushi restaurants on Dianping, Meituan, and Amap concurrently through multiple virtual screens, consolidates their ratings, and summarizes the top three options in a local report op… view at source ↗
Figure 13
Figure 13. Figure 13: Representative real-device failure patterns of Qwen 3.7 Plus. [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Representative GUI interaction patterns. The panels show (a) filling structured spreadsheet content, (b) finding content with in-page search, (c) visual navigation through scrolling, (d) use of a native media-timeline feature, (e) continuous spatial interaction, and (f) zooming in for fine-grained inspection. Each panel pairs the rendered action with its resulting application state. appear in 62.1% and 88… view at source ↗
Figure 15
Figure 15. Figure 15: Representative CLI interaction patterns. The model uses CLI tools to (a) retrieve relevant documents from a mixed corpus, (b) parse machine-readable artifact structure, (c) compress many visual candidates into one labeled comparison, (d) transform an artifact programmatically, (e) execute a repeated computation and materialize its outputs, and (f) verify exact output and process postconditions. Each panel… view at source ↗
Figure 16
Figure 16. Figure 16: Representative batched-action patterns. (a) a compact GUI macro for in-page search, (b) a 21-action dependent form sequence that stops at the next uncertain dialog, and (c) a GUI–CLI handoff that verifies a spatial manipulation through structured task state. Each row pairs the rendered batch with the first post-batch observation. more compactly and precisely as data or code than as a sequence of GUI inter… view at source ↗
Figure 17
Figure 17. Figure 17: Representative trajectory of tightly coupled GUI–CLI collaboration. The task requires Qwen-UI-Agent to achieve a score of at least 100 in a browser-based dinosaur game without using DevTools or CDP. GUI observations provide evidence for measuring jump dynamics, diagnosing failed control policies, and recognizing changes between day and night modes, while CLI code iteratively implements stateful key contro… view at source ↗
Figure 18
Figure 18. Figure 18: Action RL case study. The SFT model navigates to the account page and repeatedly clicks the share icon before navigating back, becoming trapped in an ineffective local loop. In contrast, the model after action RL recognizes that it is on the wrong page, switches to exploring Mastodon’s web interface, locates the correct entry point, and successfully configures the target invite link with the required sett… view at source ↗
Figure 19
Figure 19. Figure 19: Online RL elicits verification and self-correction. On the same task, the online RL policy (bottom) succeeds where the SFT policy (top) fails, because it reopens the application to inspect the actual result and corrects the problem it finds before finishing. The SFT policy trusts a superficial script check that only counts the chart, never looks at the rendered output, and ships an empty chart with no dat… view at source ↗
Figure 20
Figure 20. Figure 20: Online RL elicits emergent cross-modal action. On the same task, the online RL policy (bottom) succeeds where the SFT policy (top) fails, because it grounds its decisions in what it visually observes and verifies the result before finishing. The SFT policy reads the receipt from OCR text alone, mistakes a Cash Out for income, and declares success without checking, whereas the RL policy opens the receipt i… view at source ↗
Figure 21
Figure 21. Figure 21: Online RL improves long-horizon search and verification. Without online RL, the model follows incorrect entity associations and reaches the unsupported answer Guangzhou. With online RL, it identifies the key chain of Hai Yan, Nirvana in Fire, Huang Weide, and Chengdu. execution–verification transition—a state-changing Bash action followed by a read-only GUI inspection— increases from 40.2% to 52.4% [PITH… view at source ↗
Figure 22
Figure 22. Figure 22: Representative AutoJudge pass decisions. The trajectory evidence shows that the Agent successfully completes the requested task. CLICK 1 CLICK 2 CLICK 3 WAIT 4 CLICK 5 CLICK 6 CLICK 7 CLICK 8 CLICK 1 CLICK 2 CLICK 3 CLICK 4 TYPE “今天天气不错” 5 CLICK 6 CLICK 7 CLICK 8 MODEL FAILURE Query (ZH): 在朴朴超市的朴朴直播中，统计橱窗好物中的明星爆品，将价格低于10元且有货的商品加入购物车，收藏最贵的商品 Query (EN): In Pupu Supermarket's Pupu Live, review the featured … view at source ↗
Figure 23
Figure 23. Figure 23: Representative AutoJudge model-failure decisions. The Agent follows an incorrect or incomplete execution path and does not satisfy the task requirements. A.3 Additional Qualitative Examples 53 [PITH_FULL_IMAGE:figures/full_fig_p053_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Representative AutoJudge environment-error decisions. The trajectories are interrupted by live-App or execution-environment issues rather than by the Agent’s actions [PITH_FULL_IMAGE:figures/full_fig_p054_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Demonstration of long-horizon real-device GUI execution. The trajectory is rendered as key frames with the executed action annotated beneath each frame. In this 23-step shopping task, the agent decomposes the instruction into sequential sub-goals: it first empties the Taobao shopping cart, then selects a smart speaker, a smart plug, and a smart bulb from the Xiaomi flagship store within a 1,000-yuan budge… view at source ↗
Figure 26
Figure 26. Figure 26 [PITH_FULL_IMAGE:figures/full_fig_p055_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Demonstration of DeepSearch invoked on demand during GUI execution. Qwen-UI￾Agent first locates a Douyin video on evidence-based weight loss and extracts its main claims through GUI actions, then calls DeepSearch mid-trajectory to verify these claims against research papers and authoritative health sources, supplying external evidence that the interface itself does not provide. The verification conclusion… view at source ↗
Figure 28
Figure 28. Figure 28: Demonstration of proactive service based on mobile notifications. The trajectory is organized into highlighted stages: proactive task initiation from a detected notification, proactive execution phases, and a decision-ready result, with the executed action annotated beneath each key frame. A scheduled morning brief: at 08:00 the harness combines the user’s morning commitments with live weather and commute… view at source ↗
Figure 29
Figure 29. Figure 29: Demonstrations of cross-platform task execution. In this workflow, mobile subtasks run on virtual screens of the physical device, so execution does not block the user’s own actions. Receipt organization across devices: the agent locates receipt images in the photo gallery on a mobile virtual screen, transfers them to the designated PC directory, and uses GUI+CLI computer use to consolidate the receipt det… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

205 extracted references · 60 linked inside Pith

  1. [1]

    Introducing Claude Opus 4.8

    Anthropic . Introducing Claude Opus 4.8 . https://www.anthropic.com/news/claude-opus-4-8, May 2026. Accessed: 2026-07-19

    2026

  2. [2]

    Apodex-1.0 : A verification-centric agent team for discoverative intelligence

    Apodex Team . Apodex-1.0 : A verification-centric agent team for discoverative intelligence. Technical report, Apodex, 2026. URL https://www.apodex.com/blog/apodex-1.0

    2026

  3. [3]

    ^2 - Bench : Evaluating conversational agents in a dual-control environment

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. ^2 - Bench : Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. URL https://arxiv.org/abs/2506.07982

    Pith/arXiv arXiv 2025

  4. [4]

    ByteDance Seed . Seed2.0 . https://seed.bytedance.com/en/seed2, February 2026 a . Released: 2026-02-14

    2026

  5. [5]

    Seed2.1 officially released: Advancing AI productivity

    ByteDance Seed . Seed2.1 officially released: Advancing AI productivity. https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity, June 2026 b . Accessed: 2026-07-19

    2026

  6. [8]

    Gemini 3 pro, 2025

    Google DeepMind. Gemini 3 pro, 2025. URL https://deepmind.google/models/gemini/pro/

    2025

  7. [10]

    Webevolver: Enhancing web agent self-improvement with co-evolving world model

    Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webevolver: Enhancing web agent self-improvement with co-evolving world model. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 8970--8986, 2025

    2025

  8. [11]

    UI-Venus-1.5 Technical Report

    Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, Beitong Zhou, Xingran Zhou, Weizhi Chen, Sunhao Dai, Jingya Dou, Yichen Gong, Yuan Guo, Zhenlin Guo, Feng Li, Qian Li, Jinzhen Lin, Yuqi Zhou, Linchao Zhu, Liang Chen, Zhenyu Guo, Changhua Meng, and Weiqiang Wang. UI-Venus-1.5 Technical Report ....

    arXiv 2026

  9. [12]

    scrcpy: Display and control your Android device

    Genymobile . scrcpy: Display and control your Android device. https://github.com/Genymobile/scrcpy, 2018. Accessed: 2026-07-28

    2018

  10. [13]

    Project mariner

    Google . Project mariner. https://labs.google.com/mariner/landing/, 2025

    2025

  11. [14]

    Gemini 3.1 Pro Preview

    Google . Gemini 3.1 Pro Preview . https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview, February 2026 a . Google AI for Developers; accessed: 2026-07-19

    2026

  12. [15]

    Gemini 3.5 Flash

    Google . Gemini 3.5 Flash . https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash, May 2026 b . Google AI for Developers; accessed: 2026-07-19

    2026

  13. [16]

    Webvoyager: Building an end-to-end web agent with large multimodal models, 2024

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. URL https://arxiv.org/abs/2401.13919

    Pith/arXiv arXiv 2024

  14. [18]

    ToolCUA : Towards optimal GUI -tool path orchestration for computer use agents

    Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, and Jieping Ye. ToolCUA : Towards optimal GUI -tool path orchestration for computer use agents. arXiv preprint arXiv:2605.12481, 2026. URL https://arxiv.org/abs/2605.12481

    Pith/arXiv arXiv 2026

  15. [21]

    Jina reader: Converting URL s to LLM -friendly input

    Jina AI . Jina reader: Converting URL s to LLM -friendly input. https://jina.ai/reader, 2024. Accessed: 2026-07-28

    2024

  16. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European Conference on Computer Vision (ECCV), pp.\ 235--251, 2016. URL https://arxiv.org/abs/1603.07396

    Pith/arXiv arXiv 2016

  17. [23]

    Kimi K2.5 : Visual agentic intelligence

    Kimi Team . Kimi K2.5 : Visual agentic intelligence. https://www.kimi.com/blog/kimi-k2-5.html, 2026. Accessed: 2026-07-21

    2026

  18. [24]

    Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments

    Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, et al. Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2026

  19. [25]

    PhoneHarness : Harnessing phone-use agents through mixed GUI , CLI , and tool actions

    Chenxin Li, Zhengyao Fang, Zhengyang Tang, Pengyuan Lyu, Xingran Zhou, Xin Lai, Fei Tang, Liang Wu, Yiduo Guo, Weinong Wang, Junyi Li, Yi Zhang, Yang Ding, Huawen Shen, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Chengquan Zhang, and Han Hu. PhoneHarness : Harnessing phone-use agents through mixed GUI , CLI , and tool actions. arXiv pr...

    arXiv 2026

  20. [26]

    Screenspot-pro: GUI grounding for professional high-resolution computer use

    Kaixin Li, Meng Ziyang, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: GUI grounding for professional high-resolution computer use. In Workshop on Reasoning and Planning for Large Language Models, 2025 a . URL https://openreview.net/forum?id=XaKNDIAHas

    2025

  21. [28]

    SkillsBench : Benchmarking how well agent skills work across diverse tasks

    Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. SkillsBench : Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026 b . URL https://arxiv.org/abs/2602.12670

    Pith/arXiv arXiv 2026

  22. [29]

    UI-MOPD : Multi-platform on-policy distillation for continual GUI agent learning

    Niu Lian, Alan Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Yaowei Wang, Shu-Tao Xia, and Jinpeng Wang. UI-MOPD : Multi-platform on-policy distillation for continual GUI agent learning. arXiv preprint arXiv:2607.04425, 2026. URL https://arxiv.org/abs/2607.04425

    Pith/arXiv arXiv 2026

  23. [30]

    One forward beats two: Innerzoom for accurate and efficient gui grounding, 2026

    Chen Liu, Ling Chen, Hanzhang Zhou, Liangyu Chen, Chenglin Cai, Xin Yu, Steven Hoi, and Yue Wang. One forward beats two: Innerzoom for accurate and efficient gui grounding, 2026. URL https://arxiv.org/abs/2606.30084

    Pith/arXiv arXiv 2026

  24. [31]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-Bench : Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. URL https://arxiv.org/abs/2601.11868

    Pith/arXiv arXiv 2026

  25. [32]

    Playwright: Fast and reliable end-to-end testing for modern web apps

    Microsoft . Playwright: Fast and reliable end-to-end testing for modern web apps. https://playwright.dev, 2020. Accessed: 2026-07-28

    2020

  26. [33]

    MiniMax M3 : Frontier coding, 1m context, native multimodality

    MiniMax . MiniMax M3 : Frontier coding, 1m context, native multimodality. https://www.minimax.io/blog/minimax-m3, June 2026. Accessed: 2026-07-19

    2026

  27. [34]

    Kimi K2.6 : Advancing open-source coding

    Moonshot AI . Kimi K2.6 : Advancing open-source coding. https://www.kimi.com/blog/kimi-k2-6, April 2026. Accessed: 2026-07-19

    2026

  28. [35]

    Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M

    Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. Ui-vision: A desktop-centric gui benchmark for visual perception and interaction, 2025. URL https://arxiv.org/abs/2503.15661

    Pith/arXiv arXiv 2025

  29. [36]

    Introducing operator

    OpenAI . Introducing operator. https://openai.com/index/introducing-operator/, 2025

    2025

  30. [37]

    GPT-5.5 system card

    OpenAI . GPT-5.5 system card. https://deploymentsafety.openai.com/gpt-5-5, April 2026 a . Accessed: 2026-07-19

    2026

  31. [38]

    GPT-5.6 : Frontier intelligence that scales with your ambition

    OpenAI . GPT-5.6 : Frontier intelligence that scales with your ambition. https://openai.com/index/gpt-5-6/, July 2026 b . Accessed: 2026-07-20

    2026

  32. [39]

    OpenClaw : Personal ai assistant

    OpenClaw Contributors . OpenClaw : Personal ai assistant. GitHub repository, 2026. URL https://github.com/openclaw/openclaw. Accessed: 2026-07-20

    2026

  33. [40]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. BFCL-v4 leaderboard: https://gorilla.cs.b...

    2025

  34. [42]

    Qwen3.7-Plus : Multimodal agent intelligence

    Qwen Team . Qwen3.7-Plus : Multimodal agent intelligence. https://qwen.ai/blog?id=qwen3.7-plus, May 2026 a . Accessed: 2026-07-19

    2026

  35. [43]

    Qwen3.5 : Towards native multimodal agents, February 2026 b

    Qwen Team . Qwen3.5 : Towards native multimodal agents, February 2026 b . URL https://qwen.ai/blog?id=qwen3.5

    2026

  36. [44]

    QwenClawBench : Real-user-distribution benchmark for OpenClaw agents

    Qwen Team and Data Team, Alibaba Group . QwenClawBench : Real-user-distribution benchmark for OpenClaw agents. https://github.com/SKYLENAGE-AI/QwenClawBench, April 2026. Accessed: 2026-07-24

    2026

  37. [46]

    Redroid: A multi-arch, gpu-enabled android in cloud container solution

    Remote Android Project . Redroid: A multi-arch, gpu-enabled android in cloud container solution. https://github.com/remote-android/redroid-doc, 2024

    2024

  38. [47]

    Ui-tars-1.5

    ByteDance Seed. Ui-tars-1.5. https://seed-tars.com/1.5, 2025

    2025

  39. [48]

    Serper: Google search API

    Serper . Serper: Google search API . https://serper.dev, 2023. Accessed: 2026-07-28

    2023

  40. [49]

    HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

    Huawen Shen, Zhengyang Tang, Shangpin Peng, Liang Wu, Anran Zhang, Weinong Wang, Yiduo Guo, Chenxin Li, Zhengyao Fang, Yang Ding, Junyi Li, Fei Tang, Zheng Ruan, Yi Zhang, Xingran Zhou, Dingchen Yang, Sunqi Fan, Zhiyi Wan, Han Hu, Xin Lai, Pengyuan Lyu, and Chengquan Zhang. HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents . arXiv prepri...

    Pith/arXiv arXiv 2026

  41. [51]

    Phonebuddy: Training open models for agentic phone use, 2026

    Zhengyang Tang, Xin Lai, Pengyuan Lyu, Xinyuan Wang, Tianyi Bai, Chenxin Li, Yiduo Guo, Huawen Shen, Yuxuan Liu, Junyi Li, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Ji-Rong Wen, Rui Yan, Chengquan Zhang, and Han Hu. Phonebuddy: Training open models fo...

    Pith/arXiv arXiv 2026

  42. [53]

    CUA-GYM : Scaling verifiable training environments and tasks for computer-use agents

    Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, and Tao Yu. CUA-GYM : Scaling verifiable training environments and tasks for computer-use agents. arXiv preprint arXiv:2605.25624, 2026. URL https://arxiv.org/abs/2605.25624

    Pith/arXiv arXiv 2026

  43. [55]

    Measuring multimodal mathematical reasoning with MATH-Vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-Vision dataset. arXiv preprint arXiv:2402.14804, 2024 a . URL https://arxiv.org/abs/2402.14804

    Pith/arXiv arXiv 2024

  44. [56]

    Charles, Zhilin Yang, and Tao Yu

    Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu,...

    arXiv 2025

  45. [57]

    MMLU-Pro : A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro : A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024 b . URL https://arxiv.org/abs/2406.01574

    Pith/arXiv arXiv 2024

  46. [58]

    CharXiv : Charting gaps in realistic chart understanding in multimodal LLMs

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv : Charting gaps in realistic chart understanding in multimodal LLMs . arXiv preprint arXiv:2406.18521, 2024 c . URL https://arxiv.org/abs/2406.18521

    Pith/arXiv arXiv 2024

  47. [59]

    BrowseComp : A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp : A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. URL https://arxiv.org/abs/2504.12516

    Pith/arXiv arXiv 2025

  48. [60]

    Os-atlas: A foundation action model for generalist gui agents, 2024

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https://arxiv.org/abs/2410.23218

    Pith/arXiv arXiv 2024

  49. [61]

    Grok-1.5 vision preview

    xAI . Grok-1.5 vision preview. https://x.ai/news/grok-1.5v, 2024. RealWorldQA benchmark released alongside the blog post; dataset at https://huggingface.co/datasets/xai-org/RealworldQA

    2024

  50. [63]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    2024

  51. [64]

    Scaling computer-use grounding via user interface decomposition and synthesis, 2025

    Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025. URL https://arxiv.org/abs/2505.13227

    arXiv 2025

  52. [69]

    Gta1: Gui test-time scaling agent, 2025 a

    Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Caiming Xiong, and Junnan Li. Gta1: Gui test-time scaling agent, 2025 a . URL https://arxiv.org/abs/2507.05791

    Pith/arXiv arXiv 2025

  53. [71]

    Claw-Eval : Towards trustworthy evaluation of autonomous agents

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, and Tong Yang. Claw-Eval : Towards trustworthy evaluation of autonomous agents. arXiv preprint arXiv:2604.06132, 2026. URL https://arxiv.org/abs/2604.06132

    Pith/arXiv arXiv 2026

  54. [73]

    MMMU-Pro : A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-Pro : A more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1...

    Pith/arXiv arXiv 2025

  55. [74]

    GLM-4.7 : Advancing the coding capability

    Z.ai . GLM-4.7 : Advancing the coding capability. Z.ai Technical Blog, December 2025. URL https://z.ai/blog/glm-4.7. Released: 2025-12-22; accessed: 2026-07-28

    2025

  56. [76]

    MAI-UI Technical Report: Real-World Centric Foundation GUI Agents

    Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, and Steven Hoi. MAI-UI Technical Report: Real-World Centric Foundation GUI Agents . arXiv preprint arXiv:2512.22047, 2025 a . URL https://arxiv.org/abs/2512.22047

    arXiv 2025

  57. [78]

    BrowseComp-ZH : Benchmarking web browsing ability of large language models in Chinese

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. BrowseComp-ZH : Benchmarking web browsing ability of large language models in Chinese . arXiv preprint arXiv:2504.19314, 2025 b . URL https://arxiv.org/abs/2504.19314

    Pith/arXiv arXiv 2025

  58. [80]

    2025 , howpublished =

    GPT-5 Model Overview , author =. 2025 , howpublished =

    2025

  59. [81]

    2026 , howpublished =

    2026

  60. [82]

    2025 , howpublished =

    2025

  61. [83]

    2024 , eprint=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. 2024 , eprint=

    2024

  62. [84]

    arXiv preprint arXiv:2509.02544 , year=

    Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning , author=. arXiv preprint arXiv:2509.02544 , year=

    Pith/arXiv arXiv

  63. [85]

    arXiv preprint arXiv:2405.14573 , year=

    Androidworld: A dynamic benchmarking environment for autonomous agents , author=. arXiv preprint arXiv:2405.14573 , year=

    Pith/arXiv arXiv

  64. [86]

    arXiv preprint arXiv:2508.15144 , year=

    Mobile-agent-v3: Fundamental agents for gui automation , author=. arXiv preprint arXiv:2508.15144 , year=

    Pith/arXiv arXiv

  65. [87]

    2024 , eprint=

    Ponder & Press: Advancing Visual GUI Agent towards General Computer Control , author=. 2024 , eprint=

    2024

  66. [88]

    2025 , eprint=

    InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners , author=. 2025 , eprint=

    2025

  67. [89]

    2025 , eprint=

    UI-Venus Technical Report: Building High-performance UI Agents with RFT , author=. 2025 , eprint=

    2025

  68. [90]

    arXiv preprint arXiv:2507.19478 , year =

    MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents , author =. arXiv preprint arXiv:2507.19478 , year =

    Pith/arXiv arXiv

  69. [91]

    2025 , eprint=

    LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization , author=. 2025 , eprint=

    2025

  70. [92]

    2024 , eprint=

    ShowUI: One Vision-Language-Action Model for GUI Visual Agent , author=. 2024 , eprint=

    2024

  71. [93]

    2024 , eprint=

    Qwen2 Technical Report , author=. 2024 , eprint=

    2024

  72. [94]

    2024 , eprint=

    SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents , author=. 2024 , eprint=

    2024

  73. [95]

    2025 , url=

    The Showdown Computer Control Evaluation Suite , author=. 2025 , url=

    2025

  74. [96]

    2024 , eprint=

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments , author=. 2024 , eprint=

    2024

  75. [97]

    2024 , eprint=

    OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web , author=. 2024 , eprint=

    2024

  76. [98]

    2020 , eprint=

    Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements , author=. 2020 , eprint=

    2020

  77. [99]

    AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents , url=

    Chai, Yuxiang and Huang, Siyuan and Niu, Yazhe and Xiao, Han and Liu, Liang and Wang, Guozhi and Zhang, Dingyu and Ren, Shuai and Li, Hongsheng , year=. AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents , url=. doi:10.18653/v1/2025.findings-acl.110 , booktitle=

    doi:10.18653/v1/2025.findings-acl.110 2025

  78. [100]

    2025 , eprint=

    Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis , author=. 2025 , eprint=

    2025

  79. [101]

    2025 , eprint=

    Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction , author=. 2025 , eprint=

    2025

  80. [102]

    2024 , eprint=

    GPT-4o System Card , author=. 2024 , eprint=

    2024

Showing first 80 references.