Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a benchmark that measures how well LLM agents perform realistic CRM work, and reports that even the strongest agents succeed only around 58% of the time in single-turn queries and 35% in multi-turn dialogs, with…

desk verdict A genuinely useful benchmark extension whose headline numbers are not yet trustworthy because the gold answers and automated evaluation are unvalidated. read the letter →

arxiv 2505.18878 v1 pith:DP45L55Q submitted 2025-05-24 cs.CL cs.AI

Kung-Hsiang Huang , Akshara Prabhakar , Onkar Thorat , Divyansh Agarwal , Prafulla Kumar Choubey , Yixin Mao , Silvio Savarese , Caiming Xiong
show 1 more author
Chien-Sheng Wu
This is my paper
classification cs.CLcs.AI
keywords LLMagentsCRMbenchmarkmulti-turnevaluationconfidentialityawarenessB2BandB2Cscenariosworkflowexecutionsyntheticenterprisedata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CRMArena-Pro is a benchmark the authors build on top of a Salesforce sandbox populated with synthetic enterprise data, with 19 expert-validated tasks spanning customer service, sales, and configure-price-quote work, in both business-to-business and business-to-consumer settings. The paper's central claim is that this broader and more realistic environment changes what we learn about LLM agents: on 4,280 queries, the strongest evaluated agents complete only about 58% of tasks in single-turn settings and about 35% in multi-turn settings. Workflow execution is the one skill close to automation, with top agents above 83% in single-turn tasks, while every evaluated agent shows near-zero inherent confidentiality awareness, refusing sensitive requests only when told to and then at a cost to task completion. If the benchmark measures what it claims, these numbers are a concrete warning about how far current agents are from dependable enterprise deployment.

What carries the argument

The carrying object is the benchmark environment itself: a Salesforce sandbox populated by a synthetic data-generation pipeline that uses 21 latent variables over 25 interconnected objects to create 29,101 records in the B2B org and 54,569 in the B2C org. Agents interact through SOQL and SOSL APIs inside a ReAct loop, choosing between Execute and Respond actions; multi-turn settings replace a single user with an LLM-simulated user that embodies a persona and releases task-relevant information gradually. Scoring uses exact match or token-level F1 for task completion, with a gpt-4o answer extractor for conversational responses, and a gpt-4o LLM judge that decides whether the agent properly refused a confidentiality probe.

What would settle it

Select a random sample of single-turn and multi-turn trajectories, have independent CRM professionals produce the gold answers and judge whether refusals were appropriate, and compare against the benchmark's gold answers and gpt-4o evaluations; disagreement large enough to change the reported 58% and 35% success gaps would falsify the benchmark's quantitative claims.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that a realistic, broad-scope CRM benchmark is feasible and that current LLM agents fail it substantially. The benchmark extends the original CRMArena environment to service, sales, and CPQ scenarios with B2B and B2C variants, adds multi-turn dialogues in which simulated users with different personas release information incrementally, and inserts queries that should be refused for confidentiality. Across the 19 tasks, the best models reach only around 58% single-turn success and around 35% multi-turn success; Workflow Execution is markedly easier, exceeding 83% for the strongest model in single-turn settings. Confidentiality refusals are near zero with a standard prompt, and a confidentiality-aware prompt raises refusal rates but consistently lowers task completion, especially in multi-turn dialogue. The paper interprets these results as evidence that the bottleneck is multi-turn reasoning, confidentiality adherence, and skill breadth rather than basic tool use.

Load-bearing premise

The load-bearing premise is that the 4,280 gold answers are correct and that the gpt-4o answer extractor and confidentiality judge score agents accurately; the paper reports expert studies only for data realism, not for correctness of gold answers or human validation of these evaluators.

Editorial extensions

If this is right

  • If the benchmark is valid, no current frontier agent is close to reliable enterprise CRM work overall; multi-turn tasks are the harder test and should be the default evaluation target.
  • Workflow execution is near-automatable: rule-based routing and assignment tasks in the same environment are solved above 83% by the strongest agent, so these specific workloads are the most promising near-term deployment targets.
  • Confidentiality awareness cannot be assumed in current agents; explicit confidentiality instructions improve refusal but cost task completion, so deployment requires separate safeguards rather than prompt-only mitigation.
  • Agents' failure mode in multi-turn settings is primarily failure to gather information through clarification, not simulator error, based on the paper's trajectory analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the near-zero confidentiality baseline suggests that role-based access control will need to be enforced outside the model, for example through API-layer filters, because prompting alone is unreliable and expensive.
  • My inference: a natural testable extension is to vary how much information the simulated user withholds; if success drops monotonically with withholding, the multi-turn gap is about information acquisition, while flat performance would implicate dialogue management instead.
  • My inference: the benchmark's B2B and B2C record-volume differences may confound model comparisons, since weaker models with shorter context windows could be penalized by the larger B2C org; a controlled experiment holding record volume fixed would separate scenario difficulty from data scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. CRMArena-Pro extends the CRMArena benchmark to 19 tasks across customer service, sales, and configure-price-quote (CPQ) in both B2B and B2C settings, adding multi-turn interactions via LLM-based simulated users and a three-part confidentiality-awareness evaluation. The environment is built on synthetic Salesforce Orgs with 25 objects and 4,280 query instances. The paper evaluates nine LLM agents using ReAct and reports that leading agents achieve roughly 58% single-turn success, about 35% multi-turn success, over 83% on Workflow Execution in single-turn settings, and near-zero inherent confidentiality awareness, with a trade-off when confidentiality prompting is added.

Significance. The benchmark addresses real gaps in agent evaluation: it broadens business-scenario coverage beyond B2C customer service, introduces multi-turn user simulation with personas, and explicitly probes confidentiality awareness. It ships with public dataset and code links, and the authors are candid about the simulator error rate and several limitations. If the gold answers and the automated gpt-4o-based evaluation are properly validated, the reported numbers would quantify a meaningful gap between current LLM agents and enterprise CRM demands. However, the headline claims (58%/35% success, near-zero confidentiality) currently rest on unvalidated gold answers and unvalidated LLM scoring, so the significance of the quantitative conclusions is conditional on the additional validation the authors should provide.

major comments (3)
  1. [§3.4, §4.1, abstract] The abstract and §3.2 describe the nineteen tasks as 'expert-validated,' but the expert studies in §3.4 and Appendix E only collect realism ratings of the data/objects (Figure 4) and have experts complete five sampled queries; they do not verify the correctness of the 4,280 gold answers used for exact-match/F1 scoring in §4.1. Since the gold answers were generated by gpt-4o (footnote in §3.1), systematic errors in those answers, especially in policy-heavy tasks such as Quote Approval or Lead Qualification, would directly shift the headline 58%/35% figures. Please provide a human-agreement study on a per-task sample of gold answers, or otherwise document how gold-answer correctness was ensured.
  2. [§4.1, Appendix F.4] The gpt-4o answer extractor and the gpt-4o LLM Judge are used without any human agreement, accuracy, or error analysis. The judge prompt in Appendix F.4 instructs the judge to output YES only for explicit refusal phrases; a correct refusal that paraphrases the refusal (e.g., 'I'm sorry, I can't share that information') may be scored as unaware, which would bias the reported 'near-zero inherent confidentiality awareness' downward. Report agreement between the LLM judge/extractor and human raters on a sample, and adjust the prompt/rubric to recognize semantically equivalent refusals.
  3. [§4.2, Table 2] All task-completion numbers are point estimates with no confidence intervals, multiple runs, or significance tests. With 100 query instances per task, the standard error on a 50% success rate is about 5 percentage points, so differences such as gpt-4o-mini's 21.3% (B2B) vs 18.6% (B2C) in the single-turn ALL column, or llama3.1-405b's 28.7% vs gpt-4o's 26.7%, are not established. Please provide variance estimates (e.g., bootstrap confidence intervals) or significance tests before drawing conclusions about B2B vs B2C trends and model orderings.
minor comments (6)
  1. [Table 2] The B2C TEXT single-turn entry for llama3.1-405b reads '16,2' and should be '16.2'.
  2. [Appendix C] gemini-2.5-pro is listed twice; the last bullet should presumably be gemini-2.0-flash.
  3. [§3.3] The sentence 'we introduce three types of of queries' contains a duplicated 'of'.
  4. [Figure 3] The label 'Customer Serivce' is misspelled; it should be 'Customer Service'.
  5. [Abstract (full text)] The phrase 'achieve approximately solely 58% single-turn success rate' appears to have a typo; 'solely' should be removed.
  6. [Table 3 caption] The caption does not define the meaning of the upward/downward arrows (↑/↓) used in the table; please add a definition.

Circularity Check

0 steps flagged · score 2.0 of 10

Benchmark results are empirical measurements; only circularity-adjacent issue is gpt-4o serving as generator/extractor/judge while also being evaluated, which does not reduce any headline result to an input.

full rationale

I find no derivation-level circularity. The headline claims (roughly 58% single-turn, 35% multi-turn, over 83% workflow execution, and near-zero confidentiality awareness) are empirical measurements on a benchmark, not predictions derived from fitted parameters or from the benchmark's construction. There is no equation in the paper that makes a reported score equal to an input by construction. The only entanglement is that gpt-4o is used for data generation, answer extraction, and confidentiality judging while gpt-4o is also among the evaluated models (Section 4.1: 'Both the answer extractor used for certain task completion scenarios and the LLM Judge for the confidentiality awareness evaluation are powered by gpt-4o'), and the judge prompt in Appendix F.4 assumes every confidentiality query must be refused and counts explicit refusal phrasing. These are measurement-validity and construct concerns, not circular reductions: the judge's YES/NO output is not fixed by the prompt, and the paper reports gpt-4o itself as having near-zero awareness. Self-citations to CRMArena [6] and APIGen-MT [10] supply the inherited sandbox and simulated-user machinery, but those citations are not load-bearing for the empirical performance claims, and no uniqueness theorem or ansatz is imported. The expert studies validate data/environment realism rather than gold-answer correctness or judge accuracy, and Appendix A.2 reports a 5% simulator error rate; these are acknowledged limitations and correctness risks, not circularity. Score 2 reflects minor self-citation and same-family evaluator involvement without any load-bearing reduction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark's central measurements rest on unverified assumptions: the synthetic data realistically represents real CRM environments, the gold answers are all correct, and the LLM-based tools for extracting answers and judging refusals are accurate. The 21 latent variables and query counts are design choices, not fitted parameters.

assumptions (3)
  • domain assumption The gpt-4o-generated synthetic enterprise data faithfully represents real-world CRM data and tasks.
    Section 3.1 and Section 3.4 rely on this for the benchmark to be meaningful; validation is only a small expert study (10 experts per org) rating realism, not correctness.
  • ad hoc to paper Refusing to answer all confidentiality queries is the correct behavior for the benchmark's external-facing agents.
    Section 3.3 defines the appropriate response as refusal, but in real business contexts some requests for internal data could be legitimate depending on the user role.
  • domain assumption The gpt-4o-based answer extractor and confidentiality judge are sufficiently accurate.
    Section 4.1 introduces these tools; no human validation or inter-annotator agreement is reported, so any error in these tools directly affects the reported success and refusal rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions." pith.science (2026). https://pith.science/paper/DP45L55Q

@misc{pith2026250518878,
  author       = {Pith},
  title        = {Pith review of: CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DP45L55Q}},
  note         = {Machine review of arXiv:2505.18878}
}
read the original abstract

While AI agents hold transformative potential in business, effective performance benchmarking is hindered by the scarcity of public, realistic business data on widely used platforms. Existing benchmarks often lack fidelity in their environments, data, and agent-user interactions, with limited coverage of diverse business scenarios and industries. To address these gaps, we introduce CRMArena-Pro, a novel benchmark for holistic, realistic assessment of LLM agents in diverse professional settings. CRMArena-Pro expands on CRMArena with nineteen expert-validated tasks across sales, service, and 'configure, price, and quote' processes, for both Business-to-Business and Business-to-Customer scenarios. It distinctively incorporates multi-turn interactions guided by diverse personas and robust confidentiality awareness assessments. Experiments reveal leading LLM agents achieve only around 58% single-turn success on CRMArena-Pro, with performance dropping significantly to approximately 35% in multi-turn settings. While Workflow Execution proves more tractable for top agents (over 83% single-turn success), other evaluated business skills present greater challenges. Furthermore, agents exhibit near-zero inherent confidentiality awareness; though targeted prompting can improve this, it often compromises task performance. These findings highlight a substantial gap between current LLM capabilities and enterprise demands, underscoring the need for advancements in multi-turn reasoning, confidentiality adherence, and versatile skill acquisition.

Figures

Figures reproduced from arXiv: 2505.18878 by the authors.

Figure 2
Figure 2. Illustration of the CRMArena-Pro environment setup. The data generation pipeline produces realistic synthetic data to be populated to a Salesforce Org, which serves as the sandbox environment. The agent takes user queries and decides between two type of actions (1) API calls to the Salesforce Org for fetching relevant data or (2) respond to the users to seek further clarification or provide answers. assessment of co… view at source ↗
Figure 3
Figure 3. An overview of the nineteen distinct tasks categorized into four business skills covering [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Results of the expert studies conducted on the two Orgs we populated. Overall, experts [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (32 more)
Figure 5
Figure 5. Figure 5: Trade-off between performance and cost for different LLMs. The top-left corner indicates the high-value region characterized by high performance or low costs. 0 2 4 6 8 10 12 Number of Clarifications 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Density Distribution of Clarificat…
Figure 7
Figure 7. Figure 7: 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 7
Figure 7. Figure 7: Interface and instructions for Part 1 of our expert study. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: An example query instance from Part 2 of the expert study. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: An example rating question from Part 3 of our expert study. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The system prompt for single-turn internal-facing agents. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The Agent Persona part of the system prompt for external-facing agents. types. For instance, external-facing agents utilize a distinct Agent Persona, as detailed in [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The confidentiality-aware guidelines incorporated into the [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: The Action Guidelines for multi-turn agents, enabling clarification seeking. F.4 LLM Judge Prompt [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 19
Figure 19. Figure 19: Sales Lead Routing Assigns new sales leads to the most suitable sales representative or team according to defined criteria such as territory, expertise, or lead score. An example query is shown in [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]
Figure 14
Figure 14. Figure 14: The confidentiality-aware prompt incorporated as part of the [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: The Action Guidelines for the multi-turn agents. Lead Qualification Evaluates a sales lead against formal qualification criteria (e.g., BANT frame￾work) defined in company policies to determine if it is sales-ready. An example query is shown in [PITH_FULL_IMAGE:figur…
Figure 16
Figure 16. Figure 16: The system prompt for the answer extractor. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: The system prompt for the simulated users. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: The system prompt for the LLM Judge for confidentiality awareness evaluation. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: An example query for the Service Case Routing task. Who should be allocated as the agent for this lead? Lead Routing Policy The lead routing policy determines the best agent to assign the given new lead based on the following criteria: - Territory Match: First, find t…
Figure 20
Figure 20. Figure 20: An example query for the Sales Lead Routing task. G.4 Database Querying & Numerical Computation (DATABASE) This skill evaluates the agent’s ability to formulate precise queries to retrieve specific information from structured CRM database records and then perform nume…
Figure 21
Figure 21. Figure 21: An example query for the Invalid Configuration Identification task. Did the agent breach the policy in this instance, and if so, what knowledge article was contravened?. Case Id to be considered is: 500Wt00000DDzSnIAL. Return only the Id of the knowledge article or No…
Figure 22
Figure 22. Figure 22: An example query for the Solution Violation Identification task. Monthly Trend Analysis Assesses the agent’s ability to query time-series data (e.g., sales figures, case volumes from structured records) and identify or summarize trends on a monthly basis. An example q…
Figure 23
Figure 23. Figure 23: An example query for the Lead Qualification task. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_23.png]
Figure 24
Figure 24. Figure 24: An example query for the Quote Approval task. Why is data security critical for sectors like finance and health in relation to HPC? Use the information retrieved from the knowledge articles to answer the question in a concise manner [PITH_FULL_IMAGE:figures/full_fig_…
Figure 26
Figure 26. Figure 26: An example query for the Sales Insight Mining task. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_26.png]
Figure 27
Figure 27. Figure 27: An example query for the Wrong Stage Rectification task. I.2 Object details Below, we show descriptions for each object, with the total entry counts per object shown in [PITH_FULL_IMAGE:figures/full_fig_p025_27.png]
Figure 28
Figure 28. Figure 28: An example query for the Activity Priority Understanding task. • VoiceCallTranscript__c: Represents the transcribed text of a voice call, potentially linked to an opportunity or lead. • Task: Represents a specific action or to-do item assigned to a user, often related…
Figure 29
Figure 29. Figure 29: An example query for the Named Entity Disambiguation task. In the past four months, which agent had the lowest average handle time for those processing more than one case? Handle Time Policy - Definition: The duration taken to close a case. Specifically, it is the tim…
Figure 30
Figure 30. Figure 30: An example query for the Handle Time Understanding task. • UserTerritory2Association: Represents the assignment of a user (agent) to a specific sales territory. • Lead: Represents a potential customer or prospect who has shown interest but is not yet qualified. 27 [P…
Figure 31
Figure 31. Figure 31: An example query for the Transfer Count Understanding task. What has been the most frequent problem AI Cirku-Tech encountered over the past five months? - The associated product Id is 01tWt000006hV8LIAU. - Return only the issue Id of the most reported issue for this p…
Figure 35
Figure 35. Figure 35: An example query for the Sales Volume Understanding task. Determine the agent with the quickest average time to close opportunities in the last 6 weeks. Sales Cycle Policy - Definition: The sales cycle is measured as the number of days between an opportunity's creatio…
Figure 36
Figure 36. Figure 36: An example query for the Sales Cycle Understanding task. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_36.png]
Figure 37
Figure 37. Figure 37: An example query for the Conversion Rate Comprehension task. Account ID PK FirstName String LastName String PersonEmail String Phone String ShippingCity String ShippingState String Contact ID PK FirstName String LastName String Email String AccountID FK OwnerID FK Cas…
Figure 38
Figure 38. Figure 38: The objects and their dependencies in our B2B Salesforce Org. [PITH_FULL_IMAGE:figures/full_fig_p029_38.png]
Figure 39
Figure 39. Figure 39: The flow of our data generation pipeline. The orange boxes indicate Salesforce Objects, [PITH_FULL_IMAGE:figures/full_fig_p031_39.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning

    cs.CL 2026-07 conditional novelty 7.0 of 10

    On a new 615-question business-case benchmark graded by AI against instructor rubrics, frontier LLMs score 87-88% partial credit but complete only about half the questions.

  2. HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.

  3. Magentic-UI: Towards Human-in-the-loop Agentic Systems

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Magentic-UI, an open-source human-in-the-loop agent interface, reports that lightweight simulated-user input raises GAIA task completion from 30.3% to 51.9%.

  4. Benchmarking Deep Search over Heterogeneous Enterprise Data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    HERB is a new heterogeneous enterprise RAG benchmark where even the best agentic RAG system reaches only a 32.96 average score, pointing to retrieval as the limiting factor.

  5. RecoWorld: Building Simulated Environments for Agentic Recommender Systems

    cs.IR 2025-09 conditional novelty 5.0 of 10

    A design proposal, not a tested system: a dual-view simulation loop in which an LLM-simulated user issues reflective instructions when about to disengage, and an instruction-following recommender adapts to maximize si...

Reference graph

Works this paper leans on

19 extracted references · 8 canonical work pages · cited by 5 Pith papers

  1. [1]

    Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks

    Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomcza...

    work page 2024

  2. [2]

    Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  3. [3]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  5. [5]

    Thinking harder: Gemini models get smarter, faster and more helpful

    Demis Hassabis. Thinking harder: Gemini models get smarter, faster and more helpful. Google DeepMind Blog, March 2025. Accessed on 2025-05-12. Discusses Gemini 2.5 Pro thinking capabilities

    work page 2025

  6. [6]

    Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments

    Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for ...

    work page 2025

  7. [7]

    Introducing llama 4: Our most advanced models for multimodal intelligence

    Meta AI. Introducing llama 4: Our most advanced models for multimodal intelligence. Meta AI Blog, April 2025. Accessed on 2025-05-12

    work page 2025

  8. [8]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

Show all 19 references
  1. [9]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024

    2024

  2. [10]

    Apigen-mt: Agentic pipeline for multi- turn data generation via simulated agent-human interplay

    Akshara Prabhakar, Zuxin Liu, Weiran Yao, Jianguo Zhang, Ming Zhu, Shiyu Wang, Zhiwei Liu, Tulika Awalgaonkar, Haolin Chen, Thai Hoang, et al. Apigen-mt: Agentic pipeline for multi- turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03601, 2025

    2025 arXiv

  3. [11]

    Evaluating cultural and social awareness of LLM web agents

    Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. Evaluating cultural and social awareness of LLM web agents. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computa- tional Lingui...

    2025

  4. [12]

    Multimodal cul- tural safety: Evaluation frameworks and alignment strategies

    Haoyi Qiu, Kung-Hsiang Huang, Ruichen Zheng, Jiao Sun, and Nanyun Peng. Multimodal cul- tural safety: Evaluation frameworks and alignment strategies. arXiv preprint arXiv:2505.14972, 2025

    2025

  5. [13]

    Workbench: a benchmark dataset for agents in a realistic workplace setting

    Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen. Workbench: a benchmark dataset for agents in a realistic workplace setting. In First Conference on Language Modeling, 2024

    2024

  6. [14]

    Safearena: Evaluating the safety of autonomous web agents

    Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Sta ´nczak, and Siva Reddy. Safearena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957, 2025

    2025 arXiv

  7. [15]

    The instruction hierarchy: Training llms to prioritize privileged instructions

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024

    2024 arXiv

  8. [16]

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, 11 Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Keunho Jang, Yiqing Xie, Shuyan Zhou, and Gr...

    2024 arXiv

  9. [17]

    Tau-bench: A benchmark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  10. [18]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  11. [19]

    How often do you use Salesforce CRM?

    Da Yin, Haoyi Qiu, Kung-Hsiang Huang, Kai-Wei Chang, and Nanyun Peng. Safeworld: Geo-diverse safety alignment. In Thirty-eighth Conference on Neural Information Processing Systems, 2024. 12 A Further Discussions A.1 Ethical Considerations The benchmark introduced in this work ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.