REVIEW 20 cited by
Autonomous Evaluation and Refinement of Digital Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. We experiment with multiple evaluation models that trade off between inference cost, modularity of design, and accuracy. We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics. Finally, we use these evaluators to improve the performance of existing agents via fine-tuning and inference-time guidance. Without any additional supervision, we improve state-of-the-art performance by 29% on the popular benchmark WebArena, and achieve around 75% relative improvement in device control settings.
Forward citations
Cited by 20 Pith papers
-
StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
A locally deployed 8B reflector improves task success on three of four mobile GUI agent frameworks and beats GPT-5.2 by 11.83 points on offline AndroidWorld transition accuracy.
-
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
WebRetriever is a benchmark of 800 websites and 1,550 tasks with an automated evaluator (NavEval) achieving ~91–97% human agreement, showing current web agents succeed on only 11–37% of realistic tasks across three ev...
-
Estimating the Empowerment of Language Model Agents
EELMA estimates the mutual information between an LM agent's actions and future text states, and this 'empowerment' is shown to correlate with task performance across toy games and WebArena.
-
Reinforcement Learning for Machine Learning Engineering Agents
RL-trained Qwen2.5-3B outperforms prompted Claude-3.5-Sonnet and GPT-4o on 12 MLEBench tasks by an average of 22% and 24%, using two targeted RL modifications.
-
MobiAgent: A Systematic Framework for Customizable Mobile Agents
A full-stack mobile agent framework reports state-of-the-art task completion on its own DAG-based benchmark and 2-3x speedups from replaying recorded trajectories.
-
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
A fine-tuned LLM with a unit-by-unit action decomposition outperforms prior Diplomacy agents while using far less training data.
-
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.
-
Self-Challenging Language Model Agents
A language model agent can generate its own verifiable training tasks and improve its tool-use success rate by about 2x without human-annotated data.
-
ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
ManiTaskGen automatically generates diverse, feasible mobile manipulation tasks from any input scene, and uses them to benchmark and improve vision-language robot agents.
-
ProgRM: Build Better GUI Agents with Progress Rewards
ProgRM, a per-step progress reward model trained with LCS-based self-annotated labels, improves RL-trained GUI agent success rates on WikiHow relative to outcome reward models.
-
Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models
Evaluation Agent is an LLM-agent framework that evaluates visual generative models with a handful of samples per round, claiming a 10x time reduction while keeping conclusions within one tier of full-benchmark results...
-
HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization
HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.
-
Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
Branch-and-Browse, a tree-structured web agent with page-level action memory, reports 35.8% success and up to 40.4% less time than the cited Tree Search baseline on WebArena.
-
WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning
R1-style reinforcement learning on single-step web actions lifts open-source agents above gpt-4o on WorkArena while avoiding the reward hacking seen with dense rewards.
-
Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
An offline RL method learns a Q-function from frozen VLM features and extracts a device-control policy by imitating the best of several actions ranked by that Q-function.
-
PAFFA: Premeditated Actions For Fast Agents
PAFFA caches LLM-generated web interaction scripts into an action library so that runtime agents only retrieve and parameterize a pre-written API rather than parsing HTML step by step.
-
Reflection-Based Memory For Web navigation Agents
Reflection-Augmented Planning (ReAP) retrieves short self-reflections from past web navigation tasks and lifts held-out task success by 11 points on WebArena.
-
Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms
A survey that proposes the Generalist Virtual Agent concept and taxonomies for agent environments, tasks, perceptions, actions, models, and evaluation, concluding that real-world-like environments favor human-like int...
Discussion (0). Continue with ORCID to comment.