Pith. sign in

REVIEW 6 cited by

WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.00823 v2 pith:6VCT5D7W submitted 2024-05-01 cs.CL cs.AIcs.MA

Olly Styles , Sam Miller , Patricio Cerda-Mardini , Tanaya Guha , Victor Sanchez , Bertie Vidgen This is my paper
classification cs.CLcs.AIcs.MA
keywords workbenchagentstasksworkplaceabilityactivitiesbenchmarkbusiness
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Rule hybridization makes agent self-evolution gains attributable; the best evolved configuration reaches 67.07% against a 91.6% fully-informed oracle ceiling.

  2. Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Cost-aware stopping over ranked tool prefixes, trained on stop-versus-continue payoff gaps, beats score and sufficiency thresholds under heterogeneous tool costs.

  3. When Do Multi-Agent Systems Help? An Information Bottleneck Perspective

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Multi-agent LLM systems help exactly when the context removed by compressed relays outweighs the downstream-relevant information those relays discard.

  4. Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls

    cs.SE 2026-02 conditional novelty 6.0 of 10

    A simulated tool environment with argument validation, response synthesis, and task-state feedback improves LLM tool-call accuracy at test time.

  5. Benchmarking Deep Search over Heterogeneous Enterprise Data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    HERB is a new heterogeneous enterprise RAG benchmark where even the best agentic RAG system reaches only a 32.96 average score, pointing to retrieval as the limiting factor.

  6. Kaleidoscopic Teaming in Multi Agent Simulations

    cs.AI 2025-06

Pith tools