{"id":"3fac7f69-678f-4f2c-b5a4-5d11527d4720","arxiv_id":"2508.20453","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MCP-Bench introduces a 104-task, 250-tool MCP-based benchmark where even the strongest LLM scores 0.749, with planning and multi-server orchestration as the key weaknesses.","lead":"MCP-Bench links 20 AI models to 28 real-world tool servers with 250 tools and 104 multi-step tasks, testing tool selection, planning, and grounded answers. The best model reaches only 0.749 overall, and planning quality, not tool execution, is the main bottleneck.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judge prompt's explicit low-score default and off-benchmark validation make the headline proficiency/planning claims depend on rubric calibration rather than measured capability.","rationale":"The reader's weakest assumption correctly centers on LLM-judge validity. I partially agree. I add a concrete, falsifiable mechanism: the judge prompt explicitly instructs low scores, which makes the absolute proficiency levels (e.g., 0.749) at least partly self-imposed. The validation on a different task set does not establish that the judge behaves well on the complex 104 tasks. Additionally, the Overall Score aggregation is not defined in the text and cannot be reproduced from §5.2 and Table 3, which further weakens the headline numerical claims. The proposed test (removing the calibration sentences and comparing scores) directly settles whether the central conclusions are calibration-dependent. If scores and rankings are stable under the neutral prompt, my concern would be resolved; if not, the benchmark's claimed absolute difficulty and planning-differentiator findings need revision. The verdict stays CONDITIONAL pending this check.","tokens_in":37477,"tokens_out":13578,"duration_ms":142579,"concrete_test":"Select a random sample of 20 MCP-Bench trajectories with existing judge scores. Re-run the o4-mini judge with the same prompt, but delete the three calibration sentences: 'Default to 4–5 unless you have strong evidence for higher', 'Most real-world executions should score 4–6', and 'Scores of 8+ should be EXCEPTIONAL'. Recompute the overall score per the (currently undefined) aggregation, or compare the axis-level sub-scores directly. If the mean sub-scores shift by more than ~0.1 or the ordering among gpt-5, o3, and gpt-oss-120b changes, the reported 'far from proficient' and 'planning as key differentiator' conclusions are artifacts of the rubric's calibration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline conclusion that even the strongest agents are far from proficient (gpt-5 at 0.749) and that planning is the key differentiator rests entirely on the LLM judge scores. Two features of the judge make these conclusions fragile. First, the judge prompt (Appendix A.4) contains explicit calibration anchors: 'Default to 4–5 unless you have strong evidence for higher', 'Most real-world executions should score 4–6', and 'Scores of 8+ should be EXCEPTIONAL'. These instructions force a low prior onto all outputs, so the absolute 'far from proficient' scale is a property of the rubric, not of the agents. Second, the judge's validity is only tested in §6.4 on a separate 50-task WebSearch/Time set, not on the 104 MCP-Bench tasks; the reported human agreement (1.43/2) is moderate, and the judge (o4-mini) is the same model family that synthesized the tasks. If the judge is miscalibrated on complex multi-server tasks, the rankings and the 'planning as differentiator' claim could change. The paper also never defines how the six sub-scores are aggregated into the reported Overall Score: averaging the three axis scores per §5.2 yields 0.668 for gpt-5, not 0.749, so the headline numbers are not reproducible from the described method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MCP-Bench introduces a benchmark for LLM agents that use Model Context Protocol (MCP) servers. It comprises 104 tasks built from 28 real MCP servers and 250 tools, with tasks synthesized by an LLM pipeline that first discovers tool dependency chains, filters for solvability/utility, and rewrites instructions into fuzzy, natural-language requests. Agents are evaluated with a two-tier system: rule-based metrics for tool-name validity, schema compliance, execution success, and dependency order; and an LLM-as-a-judge rubric for task completion, tool usage, and planning effectiveness, with prompt shuffling and score averaging. Experiments over 20 LLMs report high schema compliance and execution success but substantially lower planning and task-completion scores, leading the authors to conclude that planning and reasoning are the main differentiators among current agents.","tokens_in":37883,"tokens_out":4427,"duration_ms":50647,"significance":"If the evaluation is trustworthy, MCP-Bench is a useful, more realistic complement to existing tool-use benchmarks: it covers multiple coordinated servers, fuzzy instructions, information grounding, and cross-domain orchestration, and it ships with code and data. The rule-based metrics (valid tool, schema compliance, execution success) are objective and reproducible, and the scale of the model comparison (20 models) is a strength. However, the headline leaderboard and the central 'planning is the key differentiator' claim rest on the LLM-judge scores, and the manuscript's own validation of that judge is performed on a different, simpler task set. Given that the judge prompt also contains explicit low-score calibration anchors, the absolute proficiency numbers and some comparative conclusions are currently fragile. The paper is worth publishing after the judge is validated on the actual benchmark and after the score aggregation is made transparent and reproducible.","major_comments":[{"comment":"The reported Overall Scores are not reproducible from the aggregation described in §5.2. The text says sub-dimensions are averaged within each axis and the final judgment score is the average of axis-level scores. For gpt-5, the six sub-scores in Table 3 are 0.677, 0.828, 0.767, 0.749, 0.649, 0.339; both the average of the six sub-scores and the average of the three implied axis means are 0.668, not the reported 0.749. For o3 the corresponding average is 0.625, not 0.715. Since Overall Score is the headline leaderboard quantity used for the paper's main conclusions, the exact formula must be stated and the tables/text corrected, or the discrepancy resolved.","section":"§5.2, Table 3"},{"comment":"The LLM judge is validated only on a separate 50-task WebSearch/Time set, not on the 104 MCP-Bench tasks. The reported human agreement of 1.43/2 is moderate, and the ablation shows only a modest CV reduction (16.8% to 15.1%). Meanwhile, the judge prompt in Appendix A.4 contains explicit low-score anchors: 'Default to 4–5 unless you have strong evidence for higher', 'Most real-world executions should score 4–6', and 'Scores of 8+ should be EXCEPTIONAL'. These instructions compress the score scale, so the absolute statement that even gpt-5 (0.749) is 'far from proficient' is at least partly a property of the rubric rather than a measured property of the agent. The authors should validate the judge on MCP-Bench tasks (e.g., with human annotations on a representative sample), report per-axis agreement and score distributions, and show that the main rankings and the planning-differentiator co","section":"§6.4 and Appendix A.4"},{"comment":"The same model family (o4-mini) is used both for task synthesis and as the default judge, and the judge is given the dependency analysis produced during synthesis. This is not derivation-level circularity, but it creates a shared-bias risk: if o4-mini's dependency analysis is inaccurate or favors a particular reasoning style, the judge may systematically reward models that happen to follow that style. The paper should provide a robustness check with at least one independent judge family (e.g., a Claude or Gemini model) on a subset of tasks, and report agreement across judges and across model families. It would also strengthen the benchmark to report how many of the 104 tasks were independently human-verified for solvability, rather than relying only on the LLM-based quality filter.","section":"§4.2, §5.2, Appendix A.4"}],"minor_comments":[{"comment":"Typo: 'disgarded' should be 'discarded'.","section":"§4.2"},{"comment":"The text says 'As shown in Table 1' for the ablation results; the correct reference is Table 7.","section":"§6.4"},{"comment":"The benchmark name is spelled inconsistently as 'MCP-RADER' in the introduction and 'MCP-RADAR' in the related-work section; please unify.","section":"§2 / References"},{"comment":"The table header 'Schema Understanding' combines valid-tool name and schema compliance; the relationship between these rule-based columns and the LLM-judge axes should be clarified in the caption, especially since the 'Overall Score' appears to be a different aggregation than the six listed sub-scores.","section":"§5.2 / Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript introduces a useful resource, but the two load-bearing issues—judge validation on the actual benchmark and the unexplained Overall Score aggregation—need to be resolved before the results can be taken at face value. I would not reject; both are fixable with additional analysis and a correction to the text/tables. The editor may also want to check whether the 'live servers' setup introduces temporal non-determinism that is sufficiently acknowledged in the reproducibility statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the largest MCP-based tool-use benchmark to date, with 28 servers, 250 tools, 104 tasks, fuzzy instruction variants, and multi-server workflows. Prior MCP benchmarks cover at most 9 servers and 42 tools, so the scale-up is genuine. The rule-based metrics (tool validity, schema compliance, execution success) are objective, the distractor-server setup is a nice stress, and the authors ship code and data. That part deserves credit.\n\nThe soft spots are real but not fatal. The biggest is that I could not reproduce the headline Overall Score from the described method. Section 5.2 says each axis is the average of its sub-dimensions, and the final score is the average of the three axis-level scores. Averaging the six sub-scores for gpt-5 gives 0.668, not the reported 0.749. There is no other aggregation formula stated anywhere in the paper. That is a concrete, checkable inconsistency, and it undercuts the leaderboard until fixed.\n\nSecond, the LLM judge's validity is only tested on a separate 50-task WebSearch/Time set in Section 6.4, not on the 104 MCP-Bench tasks. Human agreement there is 1.43 out of 2, which is moderate. The judge is o4-mini, the same model family that synthesized the tasks, and it is fed the exact dependency analysis from synthesis. That is not derivation-level circularity, but it is a self-referential design that deserves scrutiny.\n\nThird, and I think this is the one that most affects interpretation: the judge prompt contains explicit calibration anchors, including 'Default to 4–5 unless you have strong evidence for higher' and 'Most real-world executions should score 4–6.' That means the low absolute scores are partly a property of the rubric, not just the agents. The claim that even gpt-5 is 'far from proficient' needs to be softened; the relative ranking may still hold, but the absolute proficiency scale is largely a construct of the prompt. Also, there are no error bars, so differences like gpt-5 (0.749) vs o3 (0.715) should not be taken at face value.\n\nWho is this for? People building or evaluating tool-using agents, especially around MCP. It is a useful resource and a reasonable benchmark to build on. The paper deserves serious peer review, but it needs a revision that states the aggregation formula, validates the judge on the actual task distribution, and reinterprets the absolute scores. I would engage with it, and I would recommend sending it to review, though I would expect major revision.","headline":"MCP-Bench is a real scale-up for MCP tool-use evaluation, but the headline numbers aren't reproducible from the method as written, and the judge's low-score default makes the absolute 'far from proficient' claim hard to read.","tokens_in":38335,"tokens_out":2395,"would_cite":true,"duration_ms":27656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MCP-Bench, a benchmark of 104 multi-step tasks over 28 live MCP servers and 250 tools, and claims that even frontier LLM agents score below 0.75 overall, with long-horizon planning as the main bottleneck.","keywords":["tool-using LLM agents","Model Context Protocol","benchmark","multi-step planning","LLM-as-a-judge","tool retrieval","cross-server orchestration","agentic reasoning"],"falsifier":"Independently re-score a random sample of raw execution traces from the 104 tasks, either with human annotators using the paper's own rubrics or with a judge from a different model family, and compare the resulting model ordering to o4-mini's. Also check that each task admits a reference trajectory using only tools in its allowed set. If the ordering flips or many tasks lack a solvable trajectory, the central difficulty claim fails.","tokens_in":37448,"feed_emoji":"🛠️","tokens_out":6371,"duration_ms":69721,"temperature":0.7,"pith_summary":"This paper introduces MCP-Bench, a benchmark that connects LLM agents to 28 live MCP servers exposing 250 tools across finance, travel, science, academic search, and other domains. It generates 104 multi-step, fuzzy natural-language tasks that force agents to discover tools without being told their names, chain outputs across servers, ground answers in intermediate data, and coordinate several goals at once. The central claim is that current LLM agents are far from proficient at this kind of realistic tool use: the best overall score is 0.749, while schema compliance and basic execution are near-converged above 95% for many models. The headline distinction is planning: dependency awareness and parallelism separate the strongest models from weaker ones much more sharply than any execution metric. If MCP-Bench is right, the bottleneck in tool-using agents has moved from reading tool schemas to orchestrating them.","feed_headline":"Top AI agents top out at 0.75 on real tool tasks","feed_subtitle":"Benchmark of 104 tasks over 250 live tools shows schema use has converged; multi-step planning still separates the models.","key_machinery":"The load-bearing machinery is the combination of a genuine MCP server ecosystem and a two-tier evaluation. Each task runs against 28 production MCP servers, and 10 distractor servers are attached per instance so tool retrieval happens in a large heterogeneous space. An LLM-guided task synthesis pipeline first discovers dependency chains from tool input/output signatures, filters tasks for solvability and utility, then rewrites them into fuzzy conversational instructions that hide tool names and execution steps. Evaluation combines rule-based checks for tool name validity, schema compliance, runtime success, and dependency order with an LLM-as-a-judge rubric that scores task completion, tool","core_discovery":"The central discovery MCP-Bench is built to expose is that modern LLM agents handle the mechanics of tool use well and the orchestration poorly. Across 20 models, valid tool name rates are mostly above 95% and strong models exceed 98% schema compliance, but overall task scores top out at 0.749, and the largest model-to-model gaps appear in dependency awareness and parallelism, not execution fidelity. The paper interprets this as evidence that basic tool execution has largely converged while long-horizon planning, cross-server coordination, and evidence grounding remain the differentiators. The claim is not that agents cannot call tools; it is that they cannot reliably chain many complementar","pith_inferences":["A natural next test would be to re-score the 104 tasks with a judge from a different model family or with human gold labels, since o4-mini both synthesized the tasks and judged them, so rankings could partly reflect judge-model affinity.","Because MCP-Bench runs on live servers, tool outputs change over time; freezing a snapshot of tool responses would make leaderboards reproducible across later runs.","The paper's tool-call efficiency data suggest a Pareto frontier metric combining task score with number of rounds or calls could make efficiency a first-class evaluation axis.","The fuzzy-instruction format could connect directly to tool-retrieval ranking problems, where the task is to select the right tools from a large action space before any execution happens."],"forward_implications":["If planning is the main differentiator, progress on tool-using agents should come from dependency-aware planning and long-horizon coordination rather than better schema parsing.","Multi-server settings expose a robustness gap: weaker models degrade noticeably in dependency awareness and parallelism, while strong models stay stable, making cross-server orchestration a distinct capability to measure.","Fuzzy task descriptions that hide tool names can be reused by other MCP-based benchmarks to test tool discovery and retrieval under underspecification.","Prompt shuffling and score averaging reduce judge variance, with coefficient of variation dropping from 16.8% to 15.1%, suggesting the evaluation pipeline is more stable despite relying on an LLM judge.","The top overall score still below 0.75 indicates that no tested model comes close to mastering complex, multi-server tool use, leaving clear headroom for future agent designs."],"supporting_citations":[{"why":"Defines the Model Context Protocol whose standardized invocation schema lets 28 servers expose 250 compatible tools.","marker":"Anthropic et al., 2024"},{"why":"Supplies the ReAct-style multi-round planning-and-observation loop that MCP-Bench formalizes and runs agents through.","marker":"Yao et al., 2023"},{"why":"Represents the prior composed-API benchmark whose narrow domain coverage motivates MCP-Bench's broader server ecosystem.","marker":"Yao et al., 2025"},{"why":"ToolBench anchors API-based benchmarks with isolated functions that MCP-Bench distinguishes itself from.","marker":"Qin et al., 2024"},{"why":"BFCL v3 anchors the function-calling leaderboard style that MCP-Bench argues misses fuzzy multi-goal tasks.","marker":"Patil et al., 2025a"},{"why":"MCP-RADAR is an earlier MCP-based benchmark with few servers and short workflows that MCP-Bench extends in scale.","marker":"Gao et al., 2025"},{"why":"MCPEval limits coverage to five servers and nineteen tools, establishing the narrow-MCP-benchmark baseline.","marker":"Liu et al., 2025a"},{"why":"o4-mini is the model that synthesizes the tasks and serves as the default LLM judge, so the whole pipeline depends on its behavior.","marker":"OpenAI, 2025c"},{"why":"Documents LLM-judge sensitivity to rubric ordering, motivating the prompt-shuffling and score-averaging protocol used in MCP-Bench.","marker":"Li et al., 2025"}],"fun_headline_variants":["Tool calls converge, planning separates LLM agents","Agents master tool use; fail at multi-step orchestration","New benchmark: 98% tool compliance, 0.75 task ceiling","MCP-Bench: 250 tools, 104 tasks, one gap: planning","LLMs can call tools; they just can't plan the calls"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The rankings stand on the o4-mini judge's rubric scores being accurate measures of completion, grounding, and planning across the 104 tasks, but the judge was validated only on a separate 50-task set and belongs to the same model family that generated the tasks.","fun_headline_variants_meta":{"raw":{"variants":["Tool calls converge, planning separates LLM agents","Agents master tool use; fail at multi-step orchestration","New benchmark: 98% tool compliance, 0.75 task ceiling","MCP-Bench: 250 tools, 104 tasks, one gap: planning","LLMs can call tools; they just can't plan the calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000872,"raw_usage":{"total_tokens":3623,"prompt_tokens":764,"completion_tokens":2859,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2767}},"tokens_in":508,"tokens_out":2859,"duration_ms":21055,"temperature":1.0,"reasoning_tokens":2767,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:04:14.095390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-score a random sample of raw execution traces from the 104 tasks, either with human annotators using the paper's own rubrics or with a judge from a different model family, and compare the resulting model ordering to o4-mini's. Also check that each task admits a reference trajectory using only tools in its allowed set. If the ordering flips or many tasks lack a solvable trajectory, the central difficulty claim fails.","supporting_citations":[],"review_version":1}