REVIEW 12 cited by
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.
Forward citations
Cited by 12 Pith papers
-
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
VLM judges of CUA trajectories are systematically lenient; OSReward measures this with human gold, and OS-Shepherd open models close most of the reliability gap cheaply.
-
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.
-
PG-Agent: An Agent Powered by Page Graph
An MLLM GUI agent that stores past episodes as a page graph and retrieves action guidelines from it improves step success on three benchmarks.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
Magentic-UI: Towards Human-in-the-loop Agentic Systems
Magentic-UI, an open-source human-in-the-loop agent interface, reports that lightweight simulated-user input raises GAIA task completion from 30.3% to 51.9%.
-
MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them
A new benchmark with a three-way hallucination taxonomy, snapshot-based test cases, and an LLM judge shows LLM agents hallucinate at over 30% of risky decision points, with open and closed models closer than expected.
-
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.
-
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
StarDojo is a 1,000-task benchmark in Stardew Valley combining production and social activities, and the best tested MLLM (GPT-4.1) achieves only 12.7% success on its 100-task subset.
-
HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization
HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.
-
OAgents: An Empirical Study of Building Effective Agents
OAgents reports a framework and ablation study claiming open-source state-of-the-art on GAIA, but the empirical evidence is weakened by benchmark fitting and missing statistics.
-
Scaling Test-time Compute for LLM Agents
On the GAIA benchmark, Best-of-N sampling with list-wise answer selection gives the largest agent gains, and selective (score-triggered) reflection beats reflection at every step.
-
AppAgent-Claw: CLI Is All You Need for GUI Automation
A record-once, replay-many system converts demonstrated GUI workflows into reliable OpenClaw skills via layered visual localization and post-action validation, without runtime LLM inference.
Discussion (0). Sign in to comment.