REVIEW 10 cited by
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task. However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs. In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD). A GUI-TOD system can directly perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. Furthermore, we release META-GUI, a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. We also propose a multi-model action prediction and response model, which show promising results on META-GUI. The dataset, codes and leaderboard are publicly available.
Forward citations
Cited by 10 Pith papers
-
MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
MobiBench reaches near-human offline evaluation fidelity for mobile GUI agents by accepting any valid action at each step, and enables modular attribution of performance to agent components.
-
MobileRAG: Enhancing Mobile Agent with Retrieval-Augmented Generation
MobileRAG adds retrieval over local apps, web search, and past successful steps to a mobile agent, and reports a 10.3% relative gain in task success rate on a new 80-task benchmark.
-
Morae: Proactively Pausing UI Agents for User Choices
Morae, a UI agent that proactively pauses at ambiguous decision points, helps blind and low-vision users complete more tasks and express preferences better than fully autonomous agents.
-
Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
Explicitly storing short-term and long-term text memories, instead of raw screenshots, improves GUI agent accuracy on cross-app tasks, and a new annotated dataset helps 7B models approach 72B-level memory use.
-
GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
A new benchmark with 5,318 GUI tasks, including 200 abnormal ones, shows that state-of-the-art GUI agents degrade sharply when real-world anomalies appear.
-
RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents
Multimodal computer-use agents show risky intent in about 85% of tasks and complete risky actions in about 60%, measured on RiOSWorld, a new 492-task virtual-machine benchmark.
-
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.
-
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.
-
VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
The paper promises VLM-3D but the body text is entirely the MVISU-Bench mobile-agent benchmark paper, so the stated result is unsupported.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Sign in to comment.