{"id":"2339afb7-bdc9-476a-bbf3-f2a031bb6f9b","arxiv_id":"2508.12566","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A new MCP benchmark across six LLMs finds that proactive tool use is rare on first prompts, instructed tool use mainly improves in two-turn dialogues, MCP context degrades accuracy by about 9.5%, and input-token overhead rises up to 236.5x.","lead":"The authors built a benchmark to test whether six commercial large language models actually benefit from the Model Context Protocol, a standard for letting AI call external tools. They report that models rarely use tools on a first prompt, added tool context often lowers accuracy, and token costs rise sharply.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Effectiveness comparison may compare tool-calling subset against full baseline; the 9.5% degradation could be a selection artifact.","rationale":"The reader's weakest_assumption accurately identifies the load-bearing gap. I read the paper in good faith: the framework, four dimensions, 20k API calls, and the proactivity/compliance findings are useful process measurements, and the overhead finding is directionally robust even if its exact ratios inherit the same subset ambiguity. However, Finding 3 is explicitly comparative and the quoted sentence shows the With-MCP analysis was filtered to tool-calling instances while no equivalent filter is stated for the baseline. This is not an internal inconsistency in the reported numbers, but it is an unverified comparability condition; without matched subsets the headline 'automated MCP access reduces accuracy by 9.5%' is unsupported. The proposed test (recompute baseline on S and force retrieval on a fixed instance set) would settle it. I also note the absence of released artifacts and error bars, but those are secondary to the selection problem. Because the reader already recommended REJECT and this concern confirms that rejection, the verdict should remain unchanged.","tokens_in":10889,"tokens_out":4044,"duration_ms":49982,"concrete_test":"For each model-dataset cell in Table 1, obtain instance-level logs (the paper promises release of raw data). Let S be the set of instances where the model made at least one MCP call in the With-MCP condition. Recompute Without-MCP accuracy restricted to S and compute per-instance paired differences. If the mean difference is no longer approximately -9.5%, or is within sampling noise, Finding 3 is an artifact of comparing different instance sets. If raw data are unavailable, rerun a small subset (e.g., HumanEval/MBPP) with forced retrieval on all instances versus none, holding prompts constant, to test the same pairing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Finding 3 (Effectiveness, Table 1) is the paper's central negative result: MCP integration degrades accuracy by 9.5% on average. The claim requires that With-MCP and Without-MCP accuracy are measured on the same instances. The paper does not establish this. In 'Evaluation Datasets' it says: 'we only collect and analyze the tasking solving instances where MCP calls are made by LLMs for the purpose of studying the impact of MCP calls on performance and overhead.' This restricts the With-MCP condition to instances in which the LLM actually called a tool. No sentence restricts the Without-MCP baseline to that same subset; 'Without MCP' most naturally means the full dataset without tool infrastructure. If the baseline is computed on all instances, the comparison is selection-contaminated: LLMs may choose to call tools precisely on items where their parametric answer is uncertain, and those instances would score lower even with a perfectly neutral retrieved context. The observed 9.5% average degradation therefore does not isolate the causal effect of MCP context. This is load-bearing because Finding 3 is the main evidence that MCP integration is a net negative; if the comparison sets differ, the sign and magnitude of the effect are unidentified. The same subset ambiguity also affects the overhead ratios in Table 2.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MCPGAUGE, a benchmark for evaluating large language models interacting with the Model Context Protocol (MCP), and reports a large-scale study using six commercial LLMs, 30 MCP tool suites, and 25 datasets. Four dimensions are defined: proactivity (Tool Invocation Accuracy), compliance (Instruction Following Accuracy), effectiveness (accuracy/pass@k), and overhead (input-token ratio). The central empirical claims are that two-turn dialogue increases tool invocation and compliance, that MCP integration degrades task accuracy by an average of 9.5% (Finding 3), and that input-token volume increases by 3.25x to 236.5x (Finding 4). The paper interprets these results as evidence that current MCP integration is a net negative for both accuracy and cost.","tokens_in":11190,"tokens_out":7430,"duration_ms":85293,"significance":"If the effectiveness result is correct, it is an important cautionary result for the rapidly growing MCP ecosystem, and the proposed four-dimensional evaluation framework is useful for future work. The strengths of the paper are its direct overhead measurements, its use of established benchmarks with ground truth, the formal metrics for proactivity and compliance, and the scale of the experiment (approximately 20,000 API calls). The authors also state that code and raw data will be released, which is important for a benchmark paper. However, the central comparison supporting Finding 3 appears to compare differently filtered instance sets, so the headline negative result is not currently established. The same ambiguity threatens Finding 4. These issues are fixable within the manuscript's scope by re-running or re-reporting a matched-subset analysis.","major_comments":[{"comment":"The effectiveness comparison supporting Finding 3 is not shown to be matched. The text states: 'we only collect and analyze the tasking solving instances where MCP calls are made by LLMs for the purpose of studying the impact of MCP calls on performance and overhead.' This restricts the With-MCP column of Table 1 to tool-calling instances, but the manuscript never states that the Without-MCP baseline is computed on the same instances, with the same prompt template. If the baseline covers the full benchmark while With-MCP covers a self-selected subset, the 9.5% average degradation is a selection artifact: models may invoke tools preferentially on harder items. Even if the item set is identical, the With-MCP prompts include an inserted tool-use instruction, so the comparison conflates instruction effects with retrieved-context effects. To establish Finding 3, the authors must define Withou","section":"Evaluation Datasets / Table 1"},{"comment":"The two-turn setting is described only as 'appends a follow-up query.' The actual follow-up text is not given. If the follow-up repeats, paraphrases, or hints at the tool-use instruction, then the large TIA/IFA improvements in Figures 3 and 4 may be instruction repetition rather than conversational 'warm-up' or improved compliance. Please provide the exact follow-up template and, ideally, a neutral-follow-up control to separate conversational context from re-prompting. This is necessary to support Findings 1 and 2, and it is also needed for reproducibility.","section":"Conversation-depth Settings / Figures 3-4"},{"comment":"Finding 4 is also affected by the subset ambiguity. Table 2 reports total input tokens with and without MCP, but if the two totals are computed over different numbers of instances (the With-MCP total is explicitly restricted to tool-calling instances), the overhead ratio is not a meaningful per-task cost comparison. The ratio should be computed on the same matched instances and reported per instance (e.g., median and quartiles), since token counts are heavy-tailed; a small number of very long tool responses can drive a large total. Please also clarify whether the Without-MCP token counts include the inserted tool-use instructions or only the original dataset prompts.","section":"Table 2 / Overhead"}],"minor_comments":[{"comment":"The appendix should contain the exact prompts for the 160-prompt suite, the list of 30 MCP tool suites, and the modified benchmark prompts used in the effectiveness and overhead experiments.","section":"Reproducibility"},{"comment":"The aggregate percentages (9.5%, 10.2%, 17.0%, 1.4%) are not tied to a stated aggregation formula. Table 1 contains both increases and decreases (e.g., GPT-4 MATH: 0.39 to 0.65), so the headline averages should be accompanied by per-cell differences and confidence intervals or standard errors.","section":"Results / Table 1"},{"comment":"Table 1 lists 11 dataset columns, not the 25 datasets described in the text. The mapping from the 25 datasets to the reported columns is unclear and should be reconciled.","section":"Datasets"},{"comment":"The pass@k metric is written as pass@k but k is never specified; in Table 1, no k value is given. The accuracy metric also lacks a description of how exact matching or partial credit is applied.","section":"Metrics"},{"comment":"The computation platform section states experiments are conducted on one RTX 3090, but commercial API models are evaluated. If inference was via API, the local GPU is irrelevant; if some models were local, their versions and inference settings should be specified.","section":"Experiments"},{"comment":"Minor typographical issues: 'tasking solving instances' in Evaluation Datasets, 'Llama maintains' in the Compliance section, and inconsistent spacing 'MCPGAUGE'/'MCPG AUGE'.","section":"Typos"},{"comment":"The paper says code and raw data will be open-sourced 'upon paper acceptance.' For a benchmark paper, providing access to the benchmark and data at submission time is strongly preferable, even as an anonymous supplement.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the effectiveness and overhead comparisons appear to compare a self-selected tool-calling subset against an unspecified baseline. This is the load-bearing issue for Findings 3 and 4. I recommend major revision rather than rejection because the required matched-subset analysis is feasible with the existing experimental setup and data; however, if the authors cannot provide such an analysis, the paper's central negative findings would not be publishable. I also note that the 'first comprehensive evaluation framework' claim is overstated given the existence of MCP-RADAR and MCPBench; this can be softened without affecting the scientific contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: this is a serious empirical study of MCP-augmented LLMs with a genuinely useful four-axis framework, but the headline effectiveness finding (Finding 3) has a selection problem that the paper never rules out. The with-MCP scores are computed only on instances where the LLM actually made a tool call; the without-MCP baseline is not stated to be restricted to the same subset. If the baseline covers all instances, the 9.5% average degradation is contaminated: models may call tools precisely on harder items. That is exactly the load-bearing claim, so as stated the effectiveness comparison doesn't identify the causal effect of MCP context.\n\nWhat is new and good: the four dimensions (proactivity, compliance, effectiveness, overhead) are well-chosen and the 160-prompt suite for the first two dimensions is a real addition over MCP-RADAR and MCPBench. The overhead measurements in Table 2 are direct and plausible—input-token ratios of 3.25x to 236.5x are consistent with injecting large retrieved snippets. The proactivity and compliance findings, low one-turn and much higher two-turn, are clear and internally consistent, though the paper doesn't specify the two-turn follow-up content, which matters because a follow-up that repeats or paraphrases the tool-use instruction would trivially explain the improvement.\n\nThe soft spots beyond the main flaw: no error bars or sample sizes per cell, no artifact release (the code and data are only promised upon acceptance), and the reported scale (20,000 API calls, $6,000) is not a substitute for matched evaluation. The prompt suite is also not included in the paper text. All of these are fixable, and the authors seem aware of the subset issue but don't address it in the comparison.\n\nOverall: this is a paper for people working on tool-augmented LLM evaluation and MCP agent design. The framework and the overhead/proactivity/compliance results are worth a serious referee. The effectiveness claim needs a matched-subset re-analysis or it should be reported as 'accuracy on tool-calling instances only, without a matched baseline.' I'd send it to review, but I'd expect major revision on the effectiveness section.","headline":"Useful four-axis MCP evaluation and direct overhead numbers, but the headline 9.5% accuracy degradation is not established because the with-MCP and without-MCP conditions may not be matched on the same instances.","tokens_in":11666,"tokens_out":1890,"would_cite":false,"duration_ms":22289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that giving LLMs on-demand external context via the Model Context Protocol reduces average task accuracy by 9.5% across six models and inflates input-token volume by 3.25 to 236.5 times, and introduces MCPGAUGE as a benchma","keywords":["Model Context Protocol","tool-augmented LLMs","benchmark","proactivity","instruction compliance","effectiveness evaluation","computational overhead","code generation"],"falsifier":"Re-run the effectiveness comparison on the matched subset of benchmark instances where the model actually invoked an MCP tool, computing the without-MCP baseline on exactly those same instances. If the accuracy gap disappears or reverses, the 9.5% average degradation is an artifact of selection bias; if it persists, the finding stands.","tokens_in":10785,"feed_emoji":"📉","tokens_out":5401,"duration_ms":55130,"temperature":0.7,"pith_summary":"MCPGAUGE asks whether hooking LLMs to external tools through the Model Context Protocol genuinely helps. The paper's central claim is that it does not: across six commercial LLMs, automated MCP access lowers accuracy by 9.5% on average on knowledge comprehension, general reasoning, and code generation, while inflating input-token volume by factors from 3.25 to 236.5. It also finds that models rarely invoke tools on a first query but do so after a second conversational turn, and obey explicit tool-use directives better in two-turn dialogue than one-shot. These results matter because MCP is a widely adopted standard for tool-augmented agents; if the degradation is real, current integration practice trades accuracy for cost.","feed_headline":"MCP tool access cuts LLM accuracy by 9.5 percent","feed_subtitle":"Wide benchmark of six models: retrieved context hurts reasoning and code while input tokens grow up to 236 times.","key_machinery":"The evaluation infrastructure is MCPGAUGE, built on four gauges: Tool Invocation Accuracy (TIA = autonomous correct tool calls / total tool-dependent cases), Instruction Following Accuracy (IFA = compliant calls / instruction prompts), standard Accuracy or pass@k for effectiveness, and Overhead Ratio $=T_{\\text{with-MCP}}/T_{\\text{without-MCP}}$ over input tokens. It pairs a 160-prompt suite designed to require time-sensitive or obscure knowledge with 25 ground-truth datasets. The mechanism does the work of converting MCP's promise into measurable failure modes, separating 'can invoke tools' from 'benefits from tool output.'","core_discovery":"The paper proposes four evaluation dimensions—proactivity, compliance, effectiveness, and overhead—and builds MCPGAUGE, a 160-prompt suite plus 25 established datasets, to measure how six commercial LLMs behave when given 30 MCP tool suites. Its central finding is that the assumed benefit is absent: compared with running the same models without MCP, automated MCP access lowers average accuracy by 9.5% across knowledge comprehension, general reasoning, and code generation, with code generation suffering the largest average drop (17.0%). At the same time, input-token volume grows between 3.25 and 236.5 times. The paper also reports a two-turn 'warm-up' effect: models rarely invoke tools or obe","pith_inferences":["The 9.5% figure may overstate the harm if models call tools selectively on harder instances; a matched-instance reanalysis is needed before treating the degradation as causal.","The warm-up pattern likely generalizes beyond MCP to any tool-use protocol, implying the bottleneck is the model's turn-level tool-use policy rather than the protocol itself.","A cheap intervention suggested by the overhead data is to compress or summarize retrieved snippets before injection; the paper does not test this, but its token statistics make it a natural next experiment.","If the degradation is confirmed, tool-augmented agents may need to treat external context as a candidate hypothesis rather than an authoritative input, e.g., by cross-checking it against parametric knowledge."],"forward_implications":["Single-turn prompts will not reliably trigger tool use; systems should plan for an explicit warm-up or architectural uncertainty detector rather than assume autonomous tool invocation.","Explicit one-shot tool directives are weak signals; compliance improves sharply when the instruction is reinforced in a follow-up turn.","Retrieved context often interferes with internal reasoning, so agents need filtering or gating mechanisms before injecting MCP results.","Token overhead of 3.25x to 236.5x should be a first-class design constraint; token-budget guards, relevance pruning, and caching are necessary.","Code generation is the most vulnerable domain, with pass@k dropping an average 17.0%, so tool integration for coding assistants needs special safeguards."],"supporting_citations":[{"why":"Defines the Model Context Protocol that the paper evaluates, supplying the host-client-server architecture and tool invocation model.","marker":"Anthropic 2024"},{"why":"Defines the pass@k metric used to score code-generation effectiveness in the HumanEval and MBPP tasks.","marker":"Chen et al. 2021"},{"why":"Supplies the HellaSwag datasets (ActivityNet and WikiHow) used for knowledge comprehension evaluation.","marker":"Zellers et al. 2019"},{"why":"Supplies the AGI-Eval reasoning datasets that ground the general-reasoning effectiveness comparison.","marker":"Zhong et al. 2023"},{"why":"Supplies the EvalPlus versions of HumanEval and MBPP used for code-generation evaluation.","marker":"Liu et al. 2023"},{"why":"Provides the existing MCP tool-use benchmark (MCP-RADAR) that this work contrasts by adding behavioral dimensions beyond outcome accuracy.","marker":"Gao et al. 2025"}],"fun_headline_variants":["MCP access cuts LLM accuracy 9.5%","Tool-augmented LLMs: more tokens, lower scores","MCP tools hurt code generation most: 17% drop","Input tokens balloon up to 236x with MCP","MCP augments models: less accuracy, more cost"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The finding assumes that the with-MCP and without-MCP effectiveness scores are computed on the same task instances; the paper's with-MCP scores are restricted to instances where models actually made MCP calls, and the text does not say the without-MCP baseline is restricted to that same subset.","fun_headline_variants_meta":{"raw":{"variants":["MCP access cuts LLM accuracy 9.5%","Tool-augmented LLMs: more tokens, lower scores","MCP tools hurt code generation most: 17% drop","Input tokens balloon up to 236x with MCP","MCP augments models: less accuracy, more cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2762,"prompt_tokens":746,"completion_tokens":2016,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":1932}},"tokens_in":490,"tokens_out":2016,"duration_ms":15997,"temperature":1.0,"reasoning_tokens":1932,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:26:04.253309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the effectiveness comparison on the matched subset of benchmark instances where the model actually invoked an MCP tool, computing the without-MCP baseline on exactly those same instances. If the accuracy gap disappears or reverses, the 9.5% average degradation is an artifact of selection bias; if it persists, the finding stands.","supporting_citations":[],"review_version":1}