Pith. sign in

REVIEW 16 cited by

MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2507.19478 v1 pith:XPQEDQ52 submitted 2025-07-25 cs.CV cs.CL

MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

Xuehui Wang , Zhenyu Wu , JingJing Xie , Zichen Ding , Bowen Yang , Zehao Li , Zhaoyang Liu , Qingyun Li
show 20 more authors
Xuan Dong Zhe Chen Weiyun Wang Xiangyu Zhao Jixuan Chen Haodong Duan Tianbao Xie Chenyu Yang Shiqian Su Yue Yu Yuan Huang Yiqian Liu Xiao Zhang Yanting Zhang Xiangyu Yue Weijie Su Xizhou Zhu Wei Shen Jifeng Dai Wenhai Wang
This is my paper
classification cs.CV cs.CL
keywords automationtaskmmbench-guiagentsgroundingagentbenchmarkcritical
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications

    cs.CV 2026-05 unverdicted novelty 7.0

    AndroidDaily supplies 350 verifiable tasks on 94 closed-source Android apps evaluated by GRADE (87.37% human agreement), with the strongest model achieving 62% success.

  2. What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs

    cs.CV 2026-05 conditional novelty 7.0

    GUI grounding in VLMs is bottlenecked by prefill-stage candidate selection that decoding cannot fix, so Re-Prefill uses attention to extract and re-inject target tokens for up to 4.3% gains on ScreenSpot-Pro.

  3. Weblica: Scalable and Reproducible Training Environments for Visual Web Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    Weblica scales RL training for visual web agents by building thousands of reproducible environments through HTTP caching for stable replays and LLM synthesis from real sites, yielding an 8B model that beats similar op...

  4. Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

    cs.AI 2026-05 accept novelty 7.0

    GUI-SD introduces on-policy self-distillation with visually enriched privileged context and entropy-guided weighting, outperforming GRPO and naive OPSD on six GUI grounding benchmarks while improving training efficiency.

  5. Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

    cs.AI 2026-05 unverdicted novelty 7.0

    GUI-SD is the first on-policy self-distillation framework for GUI grounding that adds privileged bounding-box context and entropy-guided weighting to outperform GRPO methods on six benchmarks in accuracy and efficiency.

  6. OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents

    cs.CL 2026-04 unverdicted novelty 7.0

    OS-SPEAR is a new evaluation toolkit that tests 22 OS agents and identifies trade-offs between efficiency and safety or robustness.

  7. WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements

    cs.SE 2026-02 unverdicted novelty 7.0

    WebTestPilot symbolizes GUI elements to infer contextual oracles for end-to-end web testing from natural language specs, reporting 99% task completion and 96% precision/recall on a new bug-injected benchmark.

  8. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  9. One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

    cs.CV 2026-06 unverdicted novelty 6.0

    InnerZoom bridges cross-layer evidence in one forward pass to achieve SOTA GUI grounding accuracy on six benchmarks while cutting latency up to 31.8% versus two-pass baselines.

  10. AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees

    cs.AI 2026-05 unverdicted novelty 6.0

    AQuaUI uses adaptive quadtrees to cut visual tokens in GUI-agent LMMs by up to 29.52% at inference time while retaining 99.06% of full-token accuracy on grounding and navigation benchmarks.

  11. InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training

    cs.CL 2026-01 conditional novelty 6.0

    InfiniteWeb auto-generates complete websites with tasks and dense-reward evaluators; training UI-TARS-1.5-7B on 600 such tasks lifts OSWorld from 24.5% to 31.4%.

  12. VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents

    cs.CL 2025-09 unverdicted novelty 6.0

    VeriOS-Agent is an OS agent that proactively queries humans in untrustworthy scenarios via a query-driven framework and three-stage training, achieving 19.72% higher step-wise success rate over baselines while preserv...

  13. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    cs.CV 2025-08 unverdicted novelty 6.0

    InternVL3.5 advances open-source multimodal models with Cascade RL for +16% reasoning gains and ViR for 4x inference speedup, with the 241B model reaching SOTA among open-source MLLMs on multimodal, reasoning, and age...

  14. Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

    cs.AI 2026-07 conditional novelty 5.5

    A real-device-centric foundation GUI agent with hybrid GUI+CLI batched actions, AutoResearch data flywheel, online RL, and a proactive harness reaches SOTA mobile and competitive desktop/web scores.

  15. HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

    cs.CV 2026-07 reject novelty 5.0

    A 3B-scale GUI agent reportedly scores 82.6% on AndroidWorld and 42% on real-device tasks, but the evidence is not independently verified and may overlap with its RL training.

  16. Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding

    cs.LG 2026-04 unverdicted novelty 5.0

    A co-evolving proposer-critic RL framework improves GUI grounding accuracy by letting the model critique its own proposals rendered on screenshots.