REVIEW 24 cited by
Cradle: Empowering Foundation Agents Towards General Computer Control
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Despite the success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Cradle can understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games, five software applications, and a comprehensive benchmark, OSWorld. Cradle is the first to enable foundation agents to follow the main storyline and complete 40-minute-long real missions in the complex AAA game Red Dead Redemption 2 (RDR2). Cradle can also create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain with a maximal weekly total profit of 87% in Dealer's Life 2. Cradle can not only operate daily software, like Chrome, Outlook, and Feishu, but also edit images and videos using Meitu and CapCut. Cradle greatly extends the reach of foundation agents by enabling the easy conversion of any software, especially complex games, into benchmarks to evaluate agents' various abilities and facilitate further data collection, thus paving the way for generalist agents.
Forward citations
Cited by 24 Pith papers
-
HumanCLAW: Can Vision-Language Models Act Through a Body?
Off-the-shelf VLMs fail closed-loop whole-body find-navigate-interact tasks (best 16.8%) because they lack embodied self-awareness, not target recognition.
-
StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
A streaming-video memory system that filters redundant frames before encoding, stores older video as latent tokens, and re-injects them when visual attention drops reaches 67.73% on StreamingBench.
-
Long-Term Memory for VLA-based Agents in Open-World Task Execution
ChemBot adds dual-layer memory and future-state asynchronous inference to VLA models, enabling better long-horizon success in chemical lab automation on collaborative robots.
-
Large Language Models are Near-Optimal Decision-Makers with a Non-Human Learning Behavior
Across uncertainty, risk, and set-shifting tasks, LLMs generally outperformed humans and neared optimality while exhibiting distinctly non-human decision-making processes.
-
BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
BIMgent, a GUI-controlling LLM agent, completes 32% of BIM building modeling tasks end-to-end, outperforming baseline computer-use agents that complete none.
-
ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World
A VLM trained with GRPO and a world-model curiosity reward explores a real desktop GUI more diversely than larger frozen models, but the diversity metric is nearly identical to its training reward.
-
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
FullFront adds a three-task benchmark for webpage design, perception, and code generation, and finds top MLLMs still fail at fine-grained layout and interaction implementation.
-
lmgame-Bench: How Good are LLMs at Playing Games?
lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.
-
G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
A 7B vision-language model trained with reinforcement learning in a new four-game environment, VLM-Gym, outperforms larger proprietary models and improves its perception and reasoning abilities together.
-
DVM: Towards Controllable LLM Agents in Social Deduction Games
DVM uses a win rate constrained decision chain reward to make a Werewolf LLM agent's actual win rate trend toward a requested target.
-
Large Action Models: From Inception to Implementation
A four-phase training pipeline converts a 7B language model into a Windows GUI action model that reaches 81.2% offline and 71.0% online task success on the authors' Word test set, beating text-only GPT-4o.
-
Software Engineering for and with GUI Agent
A survey of 336 GUI-agent papers finds rapid growth alongside weak engineering support for recovery, human oversight, maintainability, and privacy, and calls for lifecycle-centered testing and governance.
-
Mimir: A Neuro-Symbolic Memory System with Dynamic Grounding for Embodied Agents in Interactive Environments
Separating world memory from task memory and grounding each goal in recalled evidence improves embodied-agent success rates by up to 42.5 points across 13 vision-language backbones.
-
MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation
A memory-augmented LLM planner that stores and retrieves page-level summaries from past trajectories improves success rates on mobile GUI task benchmarks.
-
AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent(TEA) Protocol
AgentOrchestra, built on the TEA protocol, reports state-of-the-art GAIA and strong HLE scores by coordinating specialized sub-agents with versioned tools, environments, and self-evolution.
-
TextAtari: 100K Frames Game Playing with Language Agents
TextAtari is a text-based Atari benchmark for language agents; 7-8B LLMs stay below 10% of human scores in over 90% of tested conditions, and knowledge injection helps more than chain-of-thought.
-
Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments
MultiNet v0.2 shows that current vision-language and vision-language-action models achieve near-chance zero-shot next-action accuracy on Procgen and collapse onto a small set of preferred actions.
-
Enabling Autonomic Microservice Management through Self-Learning Agents
An LLM-agent system called ServiceOdyssey learns microservice management skills through curriculum-style self-exploration and a growing skill library, demonstrated on Sock Shop.
-
SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World
SmartAgent is a GUI agent that adds user-preference reasoning through three thought steps, but its intermediate 'underlying requirement' step does not improve item recommendation over end-to-end training.
-
AppAgent-Claw: CLI Is All You Need for GUI Automation
A record-once, replay-many system converts demonstrated GUI workflows into reliable OpenClaw skills via layered visual localization and post-action validation, without runtime LLM inference.
-
Visual Language Models as Operator Agents in the Space Domain
Vision-language models can act as spacecraft operators in the KSPDG simulator from screenshots, and fine-tuning OpenVLA on ten episodes shows preliminary promise for robotic satellite inspection.
-
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
A screenshot-only GUI agent, built from an instruction-interpreting MLLM plus a LoRA-fine-tuned Qwen2-VL locator, reports state-of-the-art grounding and task success on five GUI benchmarks.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
-
A Survey on Human-Centric LLMs
A review that sorts existing evidence on how well large language models imitate individual human skills and collective social dynamics into one taxonomy.
Discussion (0). Continue with ORCID to comment.