REVIEW 23 cited by
You Only Look at Screens: Multimodal Chain-of-Action Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Autonomous graphical user interface (GUI) agents aim to facilitate task automation by interacting with the user interface without manual intervention. Recent studies have investigated eliciting the capabilities of large language models (LLMs) for effective engagement in diverse environments. To align with the input-output requirement of LLMs, most existing approaches are developed under a sandbox setting where they rely on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. Consequently, those approaches often grapple with inference inefficiency and error propagation risks. To mitigate the challenges, we introduce Auto-GUI, a multimodal solution that directly interacts with the interface, bypassing the need for environment parsing or reliance on application-dependent APIs. Moreover, we propose a chain-of-action technique -- leveraging a series of intermediate previous action histories and future action plans -- to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions, spanning multi-step tasks such as application operation, web searching, and web shopping. Experimental results show that Auto-GUI achieves state-of-the-art performance with an action type prediction accuracy of 90\% and an overall action success rate of 74\%. Code is publicly available at https://github.com/cooelf/Auto-GUI.
Forward citations
Cited by 23 Pith papers
-
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
R-VLM improves GUI grounding by combining two-stage zoom-in proposals with an IoU-weighted training loss, raising accuracy by up to 13 absolute points over SeeClick.
-
Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
A decoupled, regression-free GUI grounding pipeline using frozen MLLM description generation and CLIP-based layout matching reports large gains on ScreenSpot-Pro and Mind2Web.
-
GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planning
GraphThink uses a task graph for LLM planning prompts, GRPO rewards, and plan verification, plus a scene-graph event-driven replanner, achieving SOTA ALFRED results and stronger long-horizon generalization than API LLMs.
-
MobiAgent: A Systematic Framework for Customizable Mobile Agents
A full-stack mobile agent framework reports state-of-the-art task completion on its own DAG-based benchmark and 2-3x speedups from replaying recorded trajectories.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
MONDAY automatically converts 20,000 YouTube mobile instructional videos into 313K annotated navigation frames and reports that models trained on it generalize better on an unseen mobile OS.
-
Interpretable Risk Mitigation in LLM Agent Systems
Steering LLaMA-3-8B's internal 'good faith/bad faith' feature shifts its defection probability in the iterated prisoner's dilemma by 28 percentage points.
-
Visual Test-time Scaling for GUI Agent Grounding
RegionFocus improves GUI agent grounding by error-triggered zooming into sub-regions and aggregating candidate actions with visual landmarks on the screenshot.
-
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
Few-shot human demonstrations, parsed into structured descriptions and retrieved by instruction similarity, substantially raise mobile GUI agent accuracy and success rates across multiple models.
-
Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments
A data-centric method that relabels agent trajectories with new instructions, called backward construction, improves LLM agent performance on web, code, desktop, and data-science tasks without human labeling.
-
Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
An autonomous propose-attempt-evaluate loop with a VLM judge improves zero-shot web navigation of open-source vision-language models, from 22.6% to 33.0% on WebVoyager.
-
Qwen-Audio-VAE Technical Report
A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.
-
Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
Branch-and-Browse, a tree-structured web agent with page-level action memory, reports 35.8% success and up to 40.4% less time than the cited Tree Search baseline on WebArena.
-
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.
-
Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation
GuidNav improves GUI navigation by reranking top-k VLM action candidates with a learned process reward model, yielding about 3.4% higher static action accuracy and up to around 33% relative dynamic task success gains.
-
Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
An offline RL method learns a Q-function from frozen VLM features and extracts a device-control policy by imitating the best of several actions ranked by that Q-function.
-
Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining
A GUI-controlling visual agent that uses edge-density-based cropping and a self-training loop matches or beats models trained on 10x more data.
-
SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World
SmartAgent is a GUI agent that adds user-preference reasoning through three thought steps, but its intermediate 'underlying requirement' step does not improve item recommendation over end-to-end training.
-
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
A qualitative case study of Claude 3.5 Computer Use across 20 desktop tasks shows broad but shallow capability, with notable failures in precise text selection and self-verification.
-
Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning
VSC-RL combines VLM-generated subgoals with a subgoal-conditioned AWR-style RL objective and claims improved sample efficiency over DigiRL and WebRL on AitW and WebArena-Lite.
-
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
A 2B multimodal agent trained with two-stage supervised fine-tuning and synthesized hierarchical/reflection reasoning achieves competitive results on ScreenSpot and AndroidWorld.
-
Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms
A survey that proposes the Generalist Virtual Agent concept and taxonomies for agent environments, tasks, perceptions, actions, models, and evaluation, concluding that real-world-like environments favor human-like int...
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Discussion (0). Continue with ORCID to comment.