REVIEW 7 cited by
CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM systems can automate routine processes and enhance personalized service. However, deploying and evaluating these agents is challenging due to the lack of realistic benchmarks that reflect the complexity of real-world CRM tasks. To address this issue, we introduce CRMArena, a novel benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments. Following guidance from CRM experts and industry best practices, we designed CRMArena with nine customer service tasks distributed across three personas: service agent, analyst, and manager. The benchmark includes 16 commonly used industrial objects (e.g., account, order, knowledge article, case) with high interconnectivity, along with latent variables (e.g., complaint habits, policy violations) to simulate realistic data distributions. Experimental results reveal that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting, and less than 55% even with function-calling abilities. Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments. CRMArena is an open challenge to the community: systems that can reliably complete tasks showcase direct business value in a popular work environment.
Forward citations
Cited by 7 Pith papers
-
Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
On a new 615-question business-case benchmark graded by AI against instructor rubrics, frontier LLMs score 87-88% partial credit but complete only about half the questions.
-
Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
Frontier LLMs score about 87-88% on rubric-graded, open-ended business case questions but fully satisfy every rubric criterion on fewer than half of them.
-
Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows
EntCollabBench shows that today's LLM agents still struggle with delegation, context transfer, parameter grounding, workflow closure, and decision commitment when tested in a simulated enterprise with 11 role-speciali...
-
EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments
EconWebArena is a new benchmark with 360 curated economic tasks across 82 authoritative websites for evaluating multimodal web agents on navigation, grounding, and data extraction.
-
Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
Exploratory interview study with 17 developers identifies four forms of emergent oversight work for software agents and documents situated challenges and heuristics.
-
AgentOmnia: Scaling Agentic Models for Full-Scenario Applications
Post-training an open Qwen3-30B agent on 52,361 verifiable tasks in 5,018 synthesized stateful environments lifts its average across four agent benchmarks from 22.9% to 41.7%.
-
SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking
A monthly closed-loop RL system for ranking sales leads, using position- and speed-weighted conversion rewards and a listwise GRPO-style loss, reports +7.9% NDCG@K offline and +4.7%/+8.7% conversion lift in a 160-day ...
Discussion (0). Sign in to comment.