Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MCPEval introduces a fully automated MCP-based evaluation loop and reports a universal gap between tool execution and final-answer quality.

desk verdict A useful, open-source MCP agent evaluation framework whose main empirical finding—the universal trajectory-completion gap—is not actually supported by its own numbers, and needs major revision before the results can be trusted. read the letter →

arxiv 2507.12806 v2 pith:BURG5PRH submitted 2025-07-17 cs.AI cs.CL

Zhiwei Liu , Jielin Qiu , Shiyu Wang , Jianguo Zhang , Zuxin Liu , Roshan Ram , Haolin Chen , Weiran Yao
show 4 more authors
Shelby Heinecke Silvio Savarese Huan Wang Caiming Xiong
This is my paper
classification cs.AIcs.CL
keywords LLMagentsModelContextProtocolautomatedevaluationtool-callmatchingLLM-as-a-judgeexecution-completiongaptaskgenerationandverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MCPEval, a fully automated pipeline for evaluating LLM-based agents that act through the Model Context Protocol (MCP). Instead of hand-building a static benchmark, MCPEval reads the tools exposed by MCP servers, uses an LLM to generate candidate tasks, verifies each one by having a frontier agent execute it, and then evaluates any model under test on the verified tasks by two complementary lenses: matching its tool calls to the reference trajectory and having an LLM judge rate its planning and final answer. The claim is that this removes the manual bottleneck in agent evaluation and reveals domain-specific strengths and weaknesses that simple success/failure benchmarks miss. Across five real-world domains and ten models, the paper's main empirical finding is a universal execution-completion gap: models execute tool calls well but produce weaker final outputs, a pattern MCPEval attributes to a genuine tendency in current architectures rather than an artifact. The paper further shows that smaller, tool-enhanced models can match larger ones on selected domains, which it frames as a cost-effective deployment signal.

What carries the argument

The load-bearing object is the verified ground-truth trajectory: a task description plus the exact sequence of tool calls, parameter values, and final response produced by a frontier agent, gpt-4.1, before evaluation begins. It is built in two stages: a Task-LLM proposes tasks from MCP server tool schemas, and a verification agent executes them, with failed executions triggering an update request that refines the task until it can be completed. Evaluation then runs on two rails: tool-call matching, covering Name Match, Parameter Match, and Order Match in strict and flexible versions weighted into an overall score, and an LLM-judger rubric covering trajectory aspects such as planning, execution flow, tool selection and usage, adaptability, efficiency, and context awareness, plus completion aspects such as requirement coverage, accuracy, completeness, and usefulness. The MCP standard itself is the enabling interface: one client-server protocol connects the model under test to real tools, which is what makes task generation, verification, and scoring automatable.

What would settle it

Have human experts judge a random sample of trajectories where models diverge from the gpt-4.1 reference while still completing the task: if a large share of the mismatched tool calls are functionally correct and produce acceptable final answers, then the matching scores are measuring style alignment rather than agent capability; the same check can be done computationally by regenerating ground truth with a second frontier model and comparing rankings.

Watch

Extended reading notes

Core claim

MCPEval's central claim is that a protocol-based, end-to-end automated evaluation loop can assess LLM agents more deeply than static or manually curated benchmarks. The model under test is placed as an MCP client and must complete verified tasks generated from real tool schemas. Its trajectory is scored in two ways: (1) tool-call matching, which compares tool names, parameter values, and call order against the gpt-4.1-generated ground-truth trajectory under strict and flexible protocols; and (2) an LLM-judger rubric covering planning, execution flow, tool selection and usage, adaptability, efficiency, context awareness, requirement coverage, accuracy, completeness, and usefulness. The empirical discovery is the trajectory-completion gap: trajectory scores exceed completion scores in every domain, with only o3, and to a lesser extent o4-mini, reversing the pattern, and the paper interprets this as a genuine limitation in output synthesis. The paper also states in Section 5.2 that tool-call metrics reflect alignment with gpt-4.1's calling style rather than absolute tool-calling quality, and its Limitations section concedes that synthetic data and automated verification can introduce false ground-truth labels for ambiguous tasks; that is why it pairs matching scores with LLM judging.

Load-bearing premise

The evaluation's load-bearing premise is that gpt-4.1's tool-name choices, parameter ordering, and call sequence are the correct reference for every task; a model that reaches a correct answer through a different but equally valid API style is marked down on tool-call matching no matter how good its final output is.

Editorial extensions

If this is right

  • Any MCP server can be turned into a fresh evaluation suite automatically, so new tools can be assessed without hand-written benchmarks.
  • The verified trajectories MCPEval produces can be fed back as training data, turning evaluation into an iterative improvement loop.
  • A consistent trajectory-completion gap across models and domains points training effort toward final-answer synthesis rather than more tool-execution practice.
  • Tool-call matching scores should be read as alignment with the gpt-4.1 reference style, so outcome-based LLM judging is needed to separate capability from style.
  • Smaller models can match larger ones on selected domains, making deployment choices possible from domain-level profiles rather than model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: If the execution-completion gap holds up, the most direct payoff is a training objective: use the verified trajectories as paired data for teaching models to turn correct tool executions into complete, useful final answers.
  • Editorial extension: A second ground-truth generator would separate style from competence; ranking shifts under a different reference model would quantify how much of MCPEval's tool-call scores are gpt-4.1-specific.
  • Editorial extension: The tool-naming deficits in the National Parks domain suggest that MCP server authors can improve scores by writing more descriptive tool names and parameter documentation, and MCPEval's per-domain name-match reporting makes that a testable hypothesis.
  • Editorial extension: Because tasks are synthesized from tool schemas, the benchmark may miss messy real-world intents; adding user-log-derived tasks or adversarial task generation would test how well the domain hierarchy and the execution-completion gap generalize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MCPEval, an automated evaluation framework for LLM agents that interact with Model Context Protocol (MCP) servers. Task generation is driven by a Task-LLM from tool specifications, tasks are verified by a frontier agent that produces ground-truth trajectories, and models-under-test are scored along two dimensions: tool-call matching (name, parameter, order, and a weighted overall score) and an LLM-judge rubric covering trajectory aspects (planning, execution flow, adaptability, etc.) and completion aspects (requirement coverage, accuracy, completeness, usefulness). Experiments cover ten models across five domains. The main empirical claim is that MCPEval reveals a 'universal' trajectory-completion gap, in which models execute tool-calling trajectories well but produce weaker final outputs, with O3 as a notable exception that excels at completion.

Significance. If the central finding is valid, MCPEval would be a useful open-source contribution: it automates task creation and verification, integrates with MCP servers, provides multi-level trajectory analysis, and is released as a toolkit. The scale of the reported evaluation is large, and the paper explicitly releases code for reproducibility, which I credit. However, the headline empirical claim currently rests on an unvalidated LLM judge and on ground-truth trajectories generated by a single model, gpt-4.1, and several internal numerical contradictions undermine confidence in the reported results. The significance is therefore conditional on judge validation and on reconciliation of the inconsistent statistics.

major comments (5)
  1. [§5.4, §5.6, Appendix B.2] The central claim of a 'universal' trajectory-completion gap is supported only by the unvalidated LLM judge. No human agreement, inter-judge reliability, rubric calibration, or judge-model ablation is reported; the Limitations section flags judge cost and verification bias but not judge validity. The rubric design itself creates a potential artifact: trajectory aspects (planning, execution flow) can be satisfied by emitting a plausible sequence of tool calls, whereas completion aspects (completeness, usefulness) require a high-quality final answer, and Section 3 states that task generation 'ensures the appropriate information for tool calls are included in the task instruction,' making the trajectory dimension easier by construction. Please provide human-validated judge scores, per-aspect calibration, and a control condition that removes the rubric-asymmetry explanation.
  2. [§5.6, Table 2] The statement in Section 5.6 that 'no points appear above the diagonal reference line' is directly contradicted by Table 2: o3 has completion scores above its trajectory scores in all five domains, and o4-mini does so in at least three domains. Since Figure 7 plots trajectory versus completion across models and domains, these points should lie above the diagonal. Please reconcile this contradiction and restrict the 'universal' claim to the aggregation in which it actually holds.
  3. [§3, §5.2] The tool-call match scores are computed against ground-truth trajectories produced by gpt-4.1, so Name Match, Param Match, and Order Match measure similarity to gpt-4.1's calling style rather than absolute tool-calling correctness. The paper acknowledges this in Section 5.2, but then uses the resulting pattern to hypothesize that gpt-4.1-mini is a distilled version of gpt-4.1; that inference is circular because the metric is defined by gpt-4.1. Additionally, the LLM judge model is never identified, which prevents reproduction and makes it impossible to separate judge bias from model performance. Please specify the judge model and either remove the distillation hypothesis or support it with external evidence.
  4. [§4, Appendix A, Table 5, Figure 4, §C.7] The reported evaluation scale is internally inconsistent. Section 4 states 676 tasks; Appendix A states 5k trajectory records and 5k completion records; Figure 4(c) gives domain record counts summing to 4,973; Table 5 reports 10,115 individual task evaluations; and while Table 5 reports 50 model-domain combinations, Section C.7 reports 172 model-domain combinations. These numbers cannot all be correct unless they refer to different units (unique tasks, model-task evaluations, trajectories, or model-domain combinations). Please define each quantity precisely and correct the inconsistencies.
  5. [Appendix C.6] The statistical reliability claims are unsupported. The text states that 'All reported means include 95% confidence intervals,' that 'Effect Sizes: Performance differences exceed practical significance thresholds (>0.1),' and, in Section C.7.5, that 'p < 0.001 for all reported correlations,' but no confidence intervals, effect sizes, test statistics, or hypothesis-test descriptions are reported anywhere. Please either provide the actual statistical analyses or remove these claims.
minor comments (6)
  1. [§3] In Section 3, 'adopts a evaluation workflow' should be 'adopts an evaluation workflow.'
  2. [Table 4] Table 4 lists gpt-4.1 and other models that are not among the ten evaluated models in Section 4; please clarify the role of gpt-4.1 as the reference trajectory generator or otherwise explain why it appears in the version table.
  3. [Figures 4 and 9] Figures 4 and 9 appear to be identical in content; please remove the duplicate or use it for a different analysis.
  4. [Multiple sections] Model names are used inconsistently (for example, 'gpt-4o-mini' versus 'gpt4o-mini' and 'gpt-4.1-mini' versus 'gpt4.1-mini'); please standardize the notation throughout.
  5. [§5.4, §5.3, Figure 4(d)] Section 5.4 says that Finance has the smallest gap and Airbnb the largest, but Figure 4(d) and Appendix C.2 report National Parks with gap 0.007 and Finance with gap 0.008; the text and figure should be reconciled.
  6. [References] Several references use truncated author lists such as 'Chase and 1 others' and 'Roveda and 1 others'; please format all references with complete author lists or standard abbreviation.

Circularity Check

1 steps flagged · score 3.0 of 10

Tool-call scorecard is self-referential via gpt-4.1 ground truth, but the main trajectory-completion gap finding is empirical, not definitionally forced.

  1. fitted input called prediction [Section 3 (Task Verification), Section B.1 (Tool Call Criteria), Section 5.2 (Fine-grained Criteria Comparison)]
    "Since the tool call ground truth is generated from gpt-4.1 and other models are evaluated against this reference, the tool call metrics (Name Match, Param Match, Order Match) in Table 3 reflect alignment with gpt-4.1’s specific approach rather than absolute tool calling quality. Based on the observation of gpt4.1-mini achieving the highest scores in parameter matching (0.878) and order matching (0.887), we hypothesis that gpt-4.1-mini is a distilled model from gpt-4.1."

    The 'ground truth' trajectories used for Name/Param/Order matching are produced by gpt-4.1, and the match metrics are defined as agreement with that reference. Consequently, gpt-4.1-mini's top Parameter Match (0.878) and Order Match (0.887) scores restate similarity to the gpt-4.1 reference that the metric already encodes. The paper's distillation hypothesis is inferred from this same similarity, so the evidence is the construction itself, not an independent observation. The paper explicitly acknowledges the reference-dependence, so this is a local, disclosed circularity rather than a hidden central derivation.

full rationale

The central claimed result, the universal trajectory-completion gap, is an empirical output of LLM judging rather than a definitional identity: the rubric defines trajectory and completion aspects separately, and the data actually contain negative-gap models (O3 and O4-mini), which would be impossible if the gap were forced by construction. The lack of judge validation and potential rubric asymmetry are correctness and validity risks, not circularity. The only genuine circular element is the tool-call scorecard, where correctness is defined as agreement with gpt-4.1-generated ground truth; the paper discloses this in Section 5.2, and it is not load-bearing for the main gap finding. Overall, the paper is largely self-contained as an evaluation framework, with one disclosed self-referential metric and a non-load-bearing side hypothesis derived from it.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical or formal entities; it is a software system. The load-bearing assumptions are the validity of MCP as a standard, the correctness of gpt-4.1-generated ground truth, the reliability of the LLM judge, and the representativeness of synthetic tasks.

free parameters (2)
  • Tool match metric weights = name 0.4, parameter 0.4, order 0.2
    Default weighting in Appendix B.1 used to compute the Overall Score; no sensitivity analysis is reported, and the weights affect model rankings.
  • Flexible matching thresholds = parameter similarity >= 0.6; order match >= 0.5
    Chosen thresholds for partial credit in Appendix B.1; no ablation is provided to show robustness.
assumptions (4)
  • domain assumption Model Context Protocol is a valid standard for agent-tool communication
    Used as the foundation of the framework; described in Section 2 and the introduction.
  • ad hoc to paper A frontier LLM (gpt-4.1) executing tasks produces correct ground-truth trajectories
    Task verification in Section 3 depends on this; the Limitations section acknowledges that automated verification can produce false ground truth labels.
  • domain assumption LLM judge scores faithfully approximate human judgment of trajectory and output quality
    Used in the LLM Judger analysis in Section 3 and Appendix B.2; no human agreement study is reported.
  • domain assumption Synthetic tasks generated by LLMs are sufficiently representative of real-world agent workloads
    The whole evaluation corpus is synthetic, and the Limitations section states that synthetic data may not reflect real-world complexity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models." pith.science (2026). https://pith.science/paper/BURG5PRH

@misc{pith2026250712806,
  author       = {Pith},
  title        = {Pith review of: MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BURG5PRH}},
  note         = {Machine review of arXiv:2507.12806}
}
read the original abstract

The rapid rise of Large Language Models (LLMs)-based intelligent agents underscores the need for robust, scalable evaluation frameworks. Existing methods rely on static benchmarks and labor-intensive data collection, limiting practical assessment. We introduce MCPEval, an open-source Model Context Protocol (MCP)-based framework that automates end-to-end task generation and deep evaluation of LLM agents across diverse domains. MCPEval standardizes metrics, seamlessly integrates with native agent tools, and eliminates manual effort in building evaluation pipelines. Empirical results across five real-world domains show its effectiveness in revealing nuanced, domain-specific performance. We publicly release MCPEval https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and standardized LLM agent evaluation.

Figures

Figures reproduced from arXiv: 2507.12806 by the authors.

Figure 1
Figure 1. User interface of the MCPEval framework. The dashboard provides streamlined access to core functional [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Two-step MCP-based task generation workflow, including initial generation phase and verification phase. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. MCPEval evaluation workflow shows MCP client/server interaction, tool call correctness checking, LLM judger assessment, automated report generation. 4 Experiment Model Selection Our evaluation includes ten models spanning different architectures and capa￾bilities, including seven OpenAI Models: GPT￾4o, GPT-4o-mini, GPT-4.1-mini, GPT-4.1-nano, O3, O3-mini, O4-mini, and three Open-Source Models: Mistral-Small-24B, Qwe… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Domain performance analysis: (a) Domain ranking by LLM judger, (b) Trajectory vs completion comparison, (c) Task distribution, (d) Performance gaps by domain. 5.3 Domain Performance Analysis [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Performance gap analysis: (a) Overall gap dis [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Model performance analysis from tool call anlysis and llm judger. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Trajectory vs Completion performance corre [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Model performance analysis: (a) Overall ranking by LLM judger, (b) Trajectory vs completion performance [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Domain performance analysis: (a) Domain ranking by LLM judger, (b) Trajectory vs completion [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Aspect performance analysis: (a) Trajectory aspects (LLM judger), (b) Completion aspects (LLM judger), [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Performance gap analysis: (a) Overall gap distribution, (b) Model-wise gaps, (c) Domain-wise gaps, (d) [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Correlation analysis between tool call evaluation and LLM judger assessment: (a) Tool Call vs LLM [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Tool name vs parameter matching performance analysis: Scatter plot showing the correlation between [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Comprehensive performance analysis: (a) Domain performance across different evaluation metrics, (b) [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Domain performance across evaluation metrics: Line chart showing how each benchmark domain [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Summary of different tasks’ performance. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Summary of different models’ performance on the Airbnb task. [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Summary of different models’ performance on the Yahoo Finance task. [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Summary of different models’ performance on the Healthcare task. [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: Summary of different models’ performance on the Sports task. [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Summary of different models’ performance on the National Park task. [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

    cs.SE 2026-01 accept novelty 8.0 of 10

    MCP-Atlas is a new benchmark with 1000 tasks on production MCP servers that uses claim-level scoring to evaluate LLM agents on realistic multi-step tool-use competency.

  2. MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

    cs.SE 2026-01 unverdicted novelty 7.0 of 10

    MCP-Atlas introduces a benchmark of 36 real MCP servers, 220 tools, and 1,000 natural-language tasks to measure LLM tool-use competency in multi-server workflows.

Reference graph

Works this paper leans on

50 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Allganize. 2024. Alli for enterprise: On-premise llm app server & the model context protocol (mcp). Accessed: 2025-06-27

    work page 2024

  4. [4]

    Anthropic. 2024. Model context protocol. https://github.com/modelcontextprotocol/. Accessed: 2024-06-26

    work page 2024

  5. [5]

    Negin Arabzadeh, Anand A. D. J. C., Fabio F. C., P. G. Ipeirotis, Jin Z., Panos P., and Sanmi S. 2024. AgentEval 1.0: A comprehensive benchmark for evaluating autonomous agents. arXiv preprint arXiv:2401.07303

    work page Pith review arXiv 2024

  6. [6]

    Harrison Chase and 1 others. 2022. LangChain . https://github.com/langchain-ai/langchain

    work page 2022

  7. [7]

    Xiang Deng, Adrien Viguier, Xinyi Chen, C Gu, Xinyun Zhang, D Yogatama, M Dréze, C Jia, and W Wang. 2023. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems

    work page 2023

  8. [8]

    Yapei Fan, Tiezheng Lin, Yutao Zhang, Qing Yao, Bo Zhu, Yilun Li, Wenlin Qian, Xin Jiang, Wei Chen, Peiyi Cheng, and 1 others. 2024. From static to dynamic: A survey of evaluation methods for large language models. arXiv preprint arXiv:2402.04337

    work page Pith review arXiv 2024

Show all 50 references
  1. [9]

    Xuanqi Gao, Yue Wang, Jinsu Kim, Chang Lee, and Diyi Yang. 2025. MCP-RADAR : A multi-dimensional benchmark for evaluating tool use capabilities in large language models. arXiv preprint arXiv:2505.16700

    2025

  2. [10]

    Zhaolin Geng and Kewei Chang. 2025. REALM-Bench : A real-world planning benchmark for llms and multi-agent systems. arXiv preprint arXiv:2502.18836

    2025 arXiv

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  4. [12]

    Thai Hoang, Kung-Hsiang Huang, Shirley Kokane, Jianguo Zhang, Zuxin Liu, Ming Zhu, Jake Grigsby, Tian Lan, Michael S Ryoo, Chien-Sheng Wu, and 1 others. 2025. Lam simulator: Advancing data generation for large action model training via online exploration and trajectory feedbac...

    2025 arXiv

  5. [13]

    Zeshan Huang, Zihan Zeng, Keren Chen, Yihui Wei, Guilei Yuan, Jinhao Li, Jiaan Yang, Ziqi Wang, Jiateng Liu, Zexin Wei, and 1 others. 2024. A survey on evaluation of large language models as agents. arXiv preprint arXiv:2406.03456

    2024 arXiv

  6. [14]

    Zhexin Ji, Zhaofan Liu, Zihan Zhao, Fuhao Yuan, Cheng Li, Yang Lin, Pinyu Wang, Yaodong Zhang, and Jing Liu. 2024. https://aclanthology.org/2024.naacl-long.412 S afety B ench: A comprehensive benchmark to evaluate LLM s' safety . In Proceedings of the 2024 Conference of the No...

    2024

  7. [15]

    Carlos E Jimenez, John He, Shafiq Joty, and Wei-Hao Shang. 2023. Swe-bench: Can language models solve real-world software engineering problems? arXiv preprint arXiv:2310.06770

    2023 arXiv

  8. [16]

    Jing Yu Koh, Robert Gu, Hong-Lek Lee, Xuhui Zhou, Xingyu Geng, Hao Zhu, Zhengyun Li, Peiran Gu, Yin-Dong Zhang, Yi Zhang, and 1 others. 2024. VisualWebArena : A realistic and challenging benchmark for multimodal web agents. arXiv preprint arXiv:2401.13649

    2024 arXiv

  9. [17]

    Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, and 1 others. 2024. Spectool: A benchmark for characterizing errors in tool-use llms. arXiv preprint arXiv:2411.13547

    2024 arXiv

  10. [18]

    Percy Liang, Rishi Bommasani, Tony Lee, D Mada, D Hudson, E Hall, T Icard, H Adel, A Adipo, J Aina, and 1 others. 2022. Holistic evaluation of language models. In Advances in Neural Information Processing Systems

    2022

  11. [19]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yaran Xu, Zekun Wang, Ruobing Zhang, C Tan, C Xu, X Li, R Yang, and 1 others. 2023 a . Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

    2023 arXiv

  12. [20]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Rithesh Murthy, Liangwei Yang, Zuxin Liu, Tian Lan, Ming Zhu, Juntao Tan, Shirley Kokane, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. 2024 a . https://arxiv.org/abs/2410.18528 Pract: Op...

    2024 arXiv

  13. [21]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, and 1 others. 2023 b . Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. arXiv preprint arXiv:2308.05960

    2023 arXiv

  14. [22]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, and 1 others. 2024 b . Agentlite: A lightweight library for building and advancing task-oriented llm agent system. arXiv preprint arXiv:2402.15538

    2024 arXiv

  15. [23]

    Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, and 1 others. 2024 c . Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Sy...

    2024

  16. [24]

    Ed Lumer, Chang Lee, Jinsu Kim, and Yeong-Dae Kim. 2025. ScaleMCP : Dynamic and auto-synchronizing model context protocol tools for llm agents. arXiv preprint arXiv:2505.06416

    2025 arXiv

  17. [25]

    Yitong Ma, Zeyu Zhang, Zepu Lin, Ke Shu, and Chen Wang. 2024 a . AgentBoard : An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178. Accepted at NeurIPS 2024

    2024 arXiv

  18. [26]

    Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, and 1 others. 2024 b . Taco: Learning multi-modal action models with synthetic chains-of-thought-and-action. arXiv preprint arXiv:2412.05479

    2024

  19. [27]

    OpenAI. 2023. Gpt-4 technical report. https://arxiv.org/abs/2303.08774

    2023 arXiv

  20. [28]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Sys...

    2022

  21. [29]

    Yujia Qin, Shi-Cheng Cai, Y-H Liang, Y Zhang, X Zhao, Y Lin, Y-H Yao, X Deng, Z-K Li, C Dong, and 1 others. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

    2023 arXiv

  22. [30]

    Joao Roveda and 1 others. 2023. CrewAI . https://github.com/joaomdmoura/crewAI

    2023

  23. [31]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Aitor Garriga, and 1 others. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In Internatio...

    2022

  24. [32]

    Juntao Tan, Liangwei Yang, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Tulika Manoj Awalgaonkar, Jianguo Zhang, Weiran Yao, Ming Zhu, Shirley Kokane, Silvio Savarese, Huan Wang, Caiming Xiong, and Shelby Heinecke. 2025. https://arxiv.org/abs/2502.20616 Personabench: Evaluating ai m...

    2025 arXiv

  25. [33]

    Shuke Tang, Zexuan Shi, Wenhai Chen, Zhaowei Zhao, Zirui Zhuang, Guanguan Zhang, Feng Chen, and Jie Luo. 2024. MATRIX : A multi-agent reinforcement learning environment for text-based social interaction simulation. arXiv preprint arXiv:2405.02705

    2024 arXiv

  26. [34]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. GitHub repository

    2023

  27. [35]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language model with self generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis...

    2023

  28. [36]

    Colin White, Samuel Dooley, Manley Roberts, Arka Shtedritski, Chris Pochinkov, Shay Ku, Neel Jain, Siddharth Jha, Jiayi Ren, John Sleigh, and 1 others. 2024. LiveBench : A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314

    2024 arXiv

  29. [37]

    Qingyun Wu and 1 others. 2023. AutoGen : Enabling next-gen llm applications via multi-agent conversation framework. https://github.com/microsoft/autogen

    2023

  30. [38]

    Tianbao Xie, Danyang Chen, Zhao Gao, Chun-Che Hsieh, Tao Yao, Hongjin Cao, Zetian Jin, Yunwei Gao, Zhenmei Li, Yifei Shen, and 1 others. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972

    2024 arXiv

  31. [39]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Fung, Yixin Lin, Xingxin Wu, Wenfeng Li, Weiming Jiang, and 1 others. 2023. WizardLM : Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  32. [40]

    Meng Yan, Ruihang Liu, Jinsu Kim, Chang Lee, and Diyi Yang. 2025. MCPWorld : A unified benchmarking testbed for api, gui, and hybrid computer use agents. arXiv preprint arXiv:2506.07672

    2025 arXiv

  33. [41]

    Shunyu Yao, Howard Chen, John Gu, K R-K, C Y-F, Q Le, and D Song. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems

    2022

  34. [42]

    Yu, Li-Rong Sun, and 1 others

    Cheng Zhang, Quan Zhang, Zhipu Wu, Jun-Yan Li, Wen-Juan Lu, Chang-Gen Lin, Sa-Hai Wang, Bin Yu, Philip S. Yu, Li-Rong Sun, and 1 others. 2023 a . A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432

    2023 arXiv

  35. [43]

    Jia-Chen Zhang, Yitong Zhu, Zhuohao Li, Ge Wang, He Zhao, and Min-Ling Zhang. 2024 a . MMLU-Pro : A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574. Accepted at NeurIPS 2024

    2024 arXiv

  36. [44]

    Jianguo Zhang, Thai Hoang, Ming Zhu, Zuxin Liu, Shiyu Wang, Tulika Awalgaonkar, Akshara Prabhakar, Haolin Chen, Weiran Yao, Zhiwei Liu, and 1 others. 2025. Actionstudio: A lightweight framework for data and training of large action models. arXiv preprint arXiv:2503.22673

    2025 arXiv

  37. [45]

    Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, and 1 others. 2024 b . xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215

    2024 arXiv

  38. [46]

    Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, Huan Wang, Silvio Savarese, and Caiming Xiong. 2023 b . Dialogstudio: Towards richest and most diverse unified dataset collection for conversational ai. arXiv preprint arXiv:2307.10172

    2023 arXiv

  39. [47]

    Leon Zheng, Serena Kou, Neel Kumar, Hieu Ngo, Boxin Zhang, Zhaohui Wang, Percy Li, and Percy Liang. 2024. HELM S afety: Towards standardized safety evaluations of language models. arXiv preprint arXiv:2405.09340

    2024 arXiv

  40. [48]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Brooks, Eric Xing, and 1 others. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena . arXiv preprint arXiv:2306.05685

    2023 arXiv

  41. [49]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Li, Zhengyun Li, C Liu, P Gu, Y Zhang, C Li, and 1 others. 2023. WebArena : A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

    2023 arXiv

  42. [50]

    Jihan Zhu, Zhizheng Lin, Jun Gao, Zhaoxuan Zhou, Yaodong Zhang, Zhaofan Liu, Ceyao Zheng, Cheng Li, Zhaokai Wang, Zili Wang, and 1 others. 2024. A survey of ai agent evaluation: A hundred unsolved problems and a one-stop open-source library. arXiv preprint arXiv:2406.09844

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.