{"id":"dce827e7-4756-4167-a16b-5e673285d44f","arxiv_id":"2509.09734","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MCP-AgentBench is a 600-query, 33-server benchmark with an LLM-as-judge evaluator; it reports that Qwen3-235B-A22B outperforms proprietary models on MCP-mediated tasks.","lead":"This paper introduces MCP-AgentBench, a benchmark with 600 tasks across 33 MCP servers and 188 tools, plus an LLM-based judge, for measuring how well AI agents use MCP standardized tool interfaces. The evaluation finds that performance depends strongly on the agent framework and that open-source Qwen3 topped the leaderboard over proprietary models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MCP-Eval's 'specific data = tool usage' rule (Appendix E) cannot distinguish retrieved values from fabricated ones, so Table 1 pass rates may reward hallucination; a provenance-grounded judge is needed before rankings are trusted.","rationale":"The reader's CONDITIONAL verdict is supported by the text itself. Section 2.3 defines MCP-Eval as judging J = Ljudge(Q_final, A_ref, A_model, I_judge) with no observation input, and Appendix E makes the heuristic explicit: 'Specific data = Tool usage (no further proof needed)' and 'If data couldn't exist without tool usage, assume tool was used.' This is the load-bearing step because the paper's headline empirical claims—Qwen3-235B-A22B ReAct at 64.7% versus GPT-4o at 27.8%, for instance—are meaningful only if Pass means the agent actually obtained the external data via MCP. The judge cannot verify that; a model that fabricates a specific-looking time, price, or status receives the same signal as a model that retrieved it. The paper's consistency section does not repair this: human raters judged the same final answers with the same rubric, so the 91.67% agreement validates labeler agreement, not data provenance. The benchmark construction itself—33 servers, 188 tools, 600 queries, statelessness filtering, and human-in-the-loop query verification—is substantial and deserves credit; the problem is confined to the evaluation instrument used to produce Table 1. A grounded re-scoring with observation transcripts, or a no-tools control, would settle whether the reported rankings survive contact with ground truth. Since the reader already flagged this as the weakest assumption and conditioned the verdict on it, I recommend no change.","tokens_in":17201,"tokens_out":5122,"duration_ms":402800,"concrete_test":"The decisive check is a provenance-grounded re-scoring: take at least 60 logged trajectories spanning all six categories and both ReAct/TC modes, append the actual tool-observation transcript to the judge's input (or require every external-data claim in the final answer to appear in some observation), and re-rank the 10 models. If the top-level ordering changes or more than ~5% of pass/fail decisions flip, the final-answer-only judge is the source of the benchmark's differentiation; if rankings are stable, the current heuristic is adequate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MCP-AgentBench pass rates (Table 1) reliably differentiate agent proficiency in MCP-mediated tasks—requires MCP-Eval to determine whether the final answer reflects actual tool outputs. Section 2.3 and Appendix E explicitly sever this connection: the judge sees only Q_final, A_ref, and A_model, and the prompt instructs that 'Specific data = Tool usage (no further proof needed)' and 'If data couldn't exist without tool usage, assume tool was used.' Under this rubric, a model that writes 'current time: 3:12 AM' or 'price: $19.99' without ever calling a tool is indistinguishable from one that retrieved those values, provided the values are plausible. The paper's own error analysis lists hallucination as a failure mode, so the judged pass set necessarily mixes genuine tool-mediated successes with fabricated but specific-sounding answers. Because hallucination rates differ across models and frameworks—Qwen3-235B-A22B drops from 64.7% (ReAct) to 40.2% (TC) partly by 'not generating a tool call'—the ranking itself is contaminated. The human-consistency check in §3.3 does not close this gap: human raters applied the same final-answer heuristic to single-model outputs, not ground-truth tool outputs; the reported agreement validates labeler consistency, not data provenance. Without access to the observation history, MCP-Eval cannot enforce the 'external data' condition on which its validity depends.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MCP-AgentBench, a benchmark for evaluating language agents that interact with tools through the Model Context Protocol. It describes a testbed of 33 MCP servers exposing 188 tools, 600 queries in six complexity categories, and MCP-Eval, an LLM-as-a-judge metric that judges the agent's final answer against a reference answer. The empirical study evaluates ten proprietary and open models in ReAct and tool-calling modes, reporting pass rates and claiming that open-source Qwen3-235B-A22B with ReAct outperforms proprietary models while GPT-4o underperforms, and that framework choice strongly affects performance. Additional analyses cover task difficulty, token efficiency, human-judge consistency, and error types.","tokens_in":17527,"tokens_out":4122,"duration_ms":36790,"significance":"If the evaluation methodology were sound, MCP-AgentBench would fill a real gap: deployed MCP servers, a systematic complexity taxonomy, balanced query coverage, and a final-answer-based outcome metric are valuable ingredients for MCP agent evaluation. The authors deserve credit for building an operational testbed, including 188 tools, for human-in-the-loop query and reference-answer construction, and for making the judge prompt fully transparent in Appendix E. The benchmark's central claim, however, is that its pass rates reliably differentiate agents on MCP-mediated tool use, and that claim currently rests on an evaluation that cannot distinguish retrieved data from fabricated data. The reported rankings, including the headline result that Qwen3-235B-A22B outperforms all proprietary models, are therefore not yet supported.","major_comments":[{"comment":"The central measurement claim is not supported because MCP-Eval cannot verify that an agent used tools. The judge receives only Q_final, A_ref, and A_model, as formalized in Section 2.3, and Appendix E instructs it that 'Specific data = Tool usage (no further proof needed)' and 'If data couldn't exist without tool usage, assume tool was used.' Under this rubric, a model that fabricates a specific-looking value (for example, a current time or price) receives a Pass even if no tool was ever called. The paper's own error analysis in Section 3.3 lists hallucination as a failure mode, so the judged pass set necessarily mixes genuine tool-mediated successes with plausible-sounding but fabricated answers. Because hallucination rates differ across models and frameworks (e.g., Qwen3-235B-A22B drops from 64.7% ReAct to 40.2% TC partly by not generating a tool call), the rankings in Table 1 cannot be trusted as measuring tool-mediated proficiency.","section":"§2.3, Appendix E"},{"comment":"The human-consistency check does not repair the provenance gap. The 60 annotated items were judged by humans and by MCP-Eval using the same final-answer heuristic, with no access to tool outputs or execution trajectories; the reported 91.67% agreement and Cohen's Kappa of 0.734 therefore validate labeler agreement, not whether the answers reflect actual tool use. The sample is also limited to Claude 3.7 Sonnet outputs, so it does not establish the judge's validity for other models or for hallucination-prone cases. A provenance-based validation is needed, for example by rerunning a sample of agent trajectories, recording ground-truth tool observations, and checking whether the judge's Pass decisions align with cases where the agent actually called the required tools.","section":"§3.3"},{"comment":"The headline model-differentiation claims are presented without uncertainty quantification. Many adjacent pass rates in Table 1 differ by only a few percentage points (e.g., Claude 4 Sonnet TC at 58.0% versus Kimi K2 TC at 61.0%, or ReAct averages of 64.7% versus 59.8%), yet the text interprets these as clear trends and picks a single best model per subgroup with no confidence intervals, bootstrap error bars, or significance tests. Given that each category contains only 100 queries, a difference of a few queries can change the reported average substantially; the 'surprising trend' of open-source superiority should be hedged or supported with error analysis.","section":"§3.1, Table 1"},{"comment":"There is a potential evaluator/evaluated overlap that should be addressed. The judge model is o3-mini-high while o3-mini is among the evaluated models, and the queries and reference answers are generated with Claude 3.7 Sonnet, which is also evaluated. This does not by itself invalidate the benchmark, but it creates a risk that pass rates favor models whose answer style resembles the generator or judge family. A concrete control would be to rescore a random subset with an independent judge model and with a provenance-based verifier, and to report whether the ranking in Table 1 changes materially.","section":"§2.1, §3.1"}],"minor_comments":[{"comment":"The caption contains a typo: 'Model Comparsion' should be 'Model Comparison'.","section":"Table 1 caption"},{"comment":"The heading 'EV ALUATION' contains an unintended space; it should read 'EVALUATION'.","section":"Appendix E"},{"comment":"The subsection is headed 'Cohen's Kappa (MCP-Eval vs. Human Majority)' but the formula given is for Fleiss' Kappa; the heading should be corrected.","section":"Appendix D.3"},{"comment":"The sentence about randomized server selection maintaining the aggregate tool count near 128 is unclear, because the threshold mentioned is a tool limit, not a server limit; please clarify how tool-count randomization was performed.","section":"§2.2"},{"comment":"Figure 1 (BFCL vs. MCP-AgentBench) is not described in the text and its axes are not labeled; please add a full caption and at least one sentence explaining the comparison.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not state whether the benchmark queries, server configurations, judge prompts, or evaluation code will be released; for a benchmark paper, an availability statement would strongly improve reproducibility and trust. The core issue is the final-answer-only judge: if the authors can revise the evaluation to use execution traces or observation-verified evidence, the paper's central claim could become defensible. I would not recommend rejection, because the flaw is localizable to the evaluation methodology and can in principle be fixed within the scope of the paper, but the current version should not be accepted as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MCP-AgentBench is a real contribution to the MCP benchmarking stack, but the headline pass rates should not be trusted until the judge is grounded in tool outputs. The testbed (33 servers, 188 tools, 600 queries) and the six-category complexity design are genuinely useful; prior MCP benchmarks are smaller or server-focused. The construction pipeline is careful, with human verification and reference answers. If the dataset ships, it will be a common resource.\n\nThe soft spot is MCP-Eval. The judge sees only the final answer and reference, and the prompt explicitly says specific data = tool usage, no further proof needed. That severs evaluation from actual tool outputs. A model can fabricate a plausible current time, price, or API response and be marked pass. The paper's own error analysis lists hallucination as a failure mode, so the judged pass set mixes genuine retrievals with invented specifics. Because different models hallucinate at different rates, the rankings in Table 1 are contaminated. Qwen3-ReAct's lead over its TC mode is partly explained by the TC mode not making tool calls, but that defense cuts both ways: ReAct may be rewarded for confident fabrications.\n\nThe human-consistency check does not fix this. It used the same final-answer heuristic on 60 items from one model, so it measures labeler agreement, not whether the data came from tools. The judge-model overlap (o3-mini-high judging o3-mini) and reference answers generated by Claude 3.7 Sonnet, which is itself evaluated, are additional but weaker concerns.\n\nThis does not sink the benchmark as a dataset. The queries and testbed are still valuable for future work with trajectory-grounded evaluation. But the paper's central claim—that these pass rates reliably differentiate agent proficiency—is not supported as written. The fix is straightforward: judge on the observation history, check that key values in the final answer appear in tool outputs, or at minimum spot-check provenance on a held-out set.\n\nI'd send this to peer review: the resource deserves referee time and the flaw is correctable. Just don't let Table 1 out without a provenance check.","headline":"A useful MCP benchmark resource whose headline pass rates are not yet trustworthy because the judge treats specific-looking data as proof of tool use.","tokens_in":18080,"tokens_out":2023,"would_cite":true,"duration_ms":19092,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MCP-AgentBench claims that a 33-server, 188-tool testbed with 600 queries and an outcome-focused LLM judge can reliably differentiate language-agent proficiency on MCP-mediated tools.","keywords":["Model Context Protocol","MCP-AgentBench","MCP-Eval","language agents","tool-use evaluation","LLM-as-a-judge","benchmark","function calling"],"falsifier":"Run MCP-Eval on a held-out set where one agent variant is forbidden from calling tools and instead produces confident, specific-sounding answers with invented values, then compare its pass rate against the same queries answered with real tool outputs; if the fabricated variant passes at a comparable rate, the judge cannot verify data provenance and pass rates measure answer plausibility rather than tool use.","tokens_in":17016,"feed_emoji":"🧰","tokens_out":7428,"duration_ms":63727,"temperature":0.7,"pith_summary":"The paper tries to establish that MCP-AgentBench, built on 33 deployed MCP servers exposing 188 tools, is a valid standardized way to measure how well language agents complete real-world tasks through MCP-mediated tools. It contributes 600 queries in six interaction-complexity categories and MCP-Eval, an LLM judge that grades final task success rather than execution path. Using the benchmark, the paper reports that an open-source model, Qwen3-235B-A22B with a reasoning-acting loop, achieves the highest average pass rate (64.7%), ahead of all proprietary models tested, while GPT-4o underperforms significantly. A sympathetic reader should care because the field lacks a large-scale, operational benchmark for MCP agents, and this work directly tries to supply one with realistic infrastructure and outcome-based scoring.","feed_headline":"600-question benchmark grades AI agents on real MCP tools","feed_subtitle":"Its LLM judge scores final task success, and an open-source model outranks proprietary rivals.","key_machinery":"The load-bearing machinery is MCP-Eval, an LLM-as-a-judge protocol that takes the user query, the reference answer, and the agent's final answer, and returns pass/fail using a set of explicit principles: specific external data counts as conclusive evidence of tool use; sufficiency beats completeness; reasonable inference is allowed; and formatting, source attribution, and process details are ignored. The second component is the six-category query typology, crossing single-server versus multi-server scope with single, parallel, and sequential call dependency, which is what makes task difficulty graduated and the claim of reliable differentiation testable. The third component is the testbed itself: 33 curated, stateless, text-only MCP servers consolidated behind a unified invocation interface, so agents face a realistic but reproducible protocol surface.","core_discovery":"The central claim is that agent proficiency in MCP-mediated tool interactions can be rigorously assessed by measuring final task success on a diverse, operationally deployed server testbed, without requiring hand-annotated execution trajectories. The paper demonstrates the claim by building the testbed, generating and human-verifying 600 queries, and showing that MCP-Eval agrees with human expert judgments (91.67% agreement, Cohen's kappa 0.734) while producing a capability ranking of ten model setups. On its own terms, the discovery is that an outcome-oriented LLM judge, keyed to the presence of specific external data as evidence of tool use, can scale evaluation across a protocol-driven tool ecosystem and differentiate models in ways existing function-calling benchmarks may misrepresent.","pith_inferences":["Because MCP-Eval's judge never sees the actual tool outputs, a natural extension is to verify data provenance by feeding the judge the tool-call trace; reported pass rates might shift if fabricated-but-specific figures are detected.","The six-category typology could be extended to stateful servers and long-horizon tasks, which the paper explicitly excludes for reproducibility, to see whether performance gaps widen when agents must maintain state across queries.","The benchmark's server and tool coverage could double as a test of MCP interoperability itself, since query generation depends on schemas exposed by independently built servers.","A testable extension is to score agents not only on final pass/fail but on tool-selection efficiency, separating planning skill from successful recovery after mistakes."],"forward_implications":["If MCP-AgentBench captures real-world task success, its 600 queries give researchers a standard yardstick for comparing MCP agents across interaction complexity.","The reported ranking, with Qwen3-235B-A22B at 64.7% average pass rate ahead of all proprietary models, would shift expectations about which model families lead protocol-driven tool use.","The finding that model rankings flip between a reasoning-acting loop and native tool-calling mode (Qwen3 drops to 40.2% in the latter) implies framework choice must be reported and controlled in MCP evaluations.","If MCP-Eval's agreement with human judgments holds, benchmarks no longer need manually annotated execution traces, lowering the cost of scaling MCP evaluation."],"supporting_citations":[{"why":"Defines the Model Context Protocol standard that the benchmark is built to evaluate.","marker":"[4]"},{"why":"Provides the router that consolidates the 33 servers behind one unified invocation interface.","marker":"[5]"},{"why":"Supplies the LLM used to assist query generation and reference-answer annotation.","marker":"[3]"},{"why":"The designated judge model, o3-mini-high, that MCP-Eval relies on for pass/fail decisions.","marker":"[21]"},{"why":"The function-calling benchmark whose narrow scope the paper contrasts with MCP-mediated evaluation.","marker":"[34]"},{"why":"Establishes the LLM-as-a-judge methodology that MCP-Eval adapts.","marker":"[38]"},{"why":"Prior evaluation of MCP servers, distinguished because it measures servers rather than agents.","marker":"[18]"},{"why":"Earlier MCP agent benchmark constrained to limited servers and domains, which MCP-AgentBench extends.","marker":"[17]"},{"why":"Another MCP tool-use benchmark that lacks the diverse operational testbed aimed for here.","marker":"[9]"}],"fun_headline_variants":["MCP-AgentBench: 600 live tool tasks expose real agent gaps","Open-source agent outranks proprietary in MCP benchmark","Outcome-based MCP-Eval matches human judges at 91.67%","New benchmark grades agents on 188 real MCP tools","MCP-AgentBench: 33 servers, 600 tasks, reliability test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's pass rates stand or fall on the premise that an LLM judge can detect genuine tool use from the final answer alone: the evaluation prompt treats any specific external-looking data, such as a current time, price, or API value, as conclusive evidence that a tool was called, so a model that fabricates plausible specific values could be marked pass without ever touching a tool.","fun_headline_variants_meta":{"raw":{"variants":["MCP-AgentBench: 600 live tool tasks expose real agent gaps","Open-source agent outranks proprietary in MCP benchmark","Outcome-based MCP-Eval matches human judges at 91.67%","New benchmark grades agents on 188 real MCP tools","MCP-AgentBench: 33 servers, 600 tasks, reliability test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1322,"prompt_tokens":926,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":542,"tokens_out":396,"duration_ms":4082,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:00:37.681755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MCP-Eval on a held-out set where one agent variant is forbidden from calling tools and instead produces confident, specific-sounding answers with invented values, then compare its pass rate against the same queries answered with real tool outputs; if the fabricated variant passes at a comparable rate, the judge cannot verify data provenance and pass rates measure answer plausibility rather than tool use.","supporting_citations":[{"cited_title":"Patil, Ion Stoica, and Joseph E","cited_arxiv_id":null,"evidence_quote":"The function-calling benchmark whose narrow scope the paper contrasts with MCP-mediated evaluation."}],"review_version":1}