{"id":"0d638764-80c7-480b-a3cf-96c8d401fba5","arxiv_id":"2508.15760","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LiveMCP-101 tests 18 AI agents on 101 live, multi-step MCP tool tasks; no model exceeds 58.42% task success, and semantic parameter errors dominate failures.","lead":"This paper introduces LiveMCP-101, a benchmark of 101 real-world multi-step queries that force AI agents to chain together tools from 41 Model Context Protocol (MCP) servers. Even the best frontier model tested, GPT-5, succeeds on only 58.42% of tasks, and the authors catalogue seven recurring failure modes in tool planning, parameterization, and output handling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal misalignment between reference and 30-round agents may systematically penalize correct answers, threatening the sub-60% TSR headline.","rationale":"The paper's core innovation is the parallel reference-execution framework, and its headline claim depends on that framework producing a valid co-temporal gold standard. The reader identified temporal alignment as the weakest assumption, and my read agrees: the 30-round agent limit versus 5.4-call reference creates a real time gap on live backends, and the paper provides no evidence that this gap is harmless. The stability analysis and human-judge validation address reproducibility and judge quality, not the temporal validity of the reference. Since this concern is exactly the reader's weakest assumption, and the recommended test would settle it, the conditional verdict remains appropriate. I credit the benchmark's genuine strengths: validated plans, human-validated judging with high agreement, multi-run stability, and a clear failure taxonomy. But the sub-60% claim is the central result, and it is not yet robust to the temporal-misalignment critique.","tokens_in":9239,"tokens_out":8212,"duration_ms":92325,"concrete_test":"On a stratified random subset of tasks (e.g., 30 tasks, 10 per difficulty tier), re-run the validated reference plan immediately after each evaluated agent finishes, producing a reference at the agent's completion timestamp. Re-score the agent's output against this post-hoc reference using the same GPT-4.1 judge. If a meaningful fraction of originally incorrect tasks become correct (e.g., GPT-5's TSR shifts by more than 5 points or crosses 60%), temporal drift is materially biasing the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that even frontier LLMs achieve <60% TSR rests on the assumption that the GPT-4.1 reference output, produced by executing a plan averaging 5.4 tool calls, is a valid gold standard for an evaluated agent allowed up to 30 rounds (Section 4.1). No mechanism synchronizes the reference's completion with the agent's final tool call. On live, time-varying backends, data can change between these windows. The LLM judge compares the agent's final result and trajectory against the co-temporal reference; a correct answer based on the agent's own recent tool calls can be scored wrong because the reference captured older data. This would systematically deflate TSR, potentially moving GPT-5 (58.42%) across the 60% threshold. The stability runs (Table 3) only vary random seeds under identical protocol and cannot detect this bias; the human-judge study (Section 4.4) validates the judge against human raters using the same reference, not the reference's temporal validity. Thus the headline claim is insecure without evidence that reference and evaluated agents are effectively contemporaneous.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LiveMCP-101 introduces a benchmark of 101 real-world queries requiring coordinated use of tools across 41 MCP servers and 260 tools, stratified into difficulty tiers. To handle time-varying backends, the paper proposes a parallel evaluation protocol: a reference agent executes a human-validated plan (averaging 5.4 tool calls) while the evaluated agent acts autonomously, and an LLM judge scores final results and trajectories against the reference. Experiments across 18 models report GPT-5 as best with 58.42% task success rate (TSR), all open-source models below 23%, and a seven-subtype failure taxonomy. Ablations study iteration budget and MCP pool size.","tokens_in":9393,"tokens_out":7761,"duration_ms":87034,"significance":"If the co-temporal ground-truth assumption holds, this is a valuable benchmark: it targets live, time-varying backends rather than static mock environments; it couples validated execution plans with parallel reference execution; it includes human-judge agreement (κ > 0.85 for results, > 0.78 for trajectories on a 30-task subset), three-run stability, and three-judge consistency; and its failure taxonomy is actionable. The headline result—that even frontier LLMs remain below 60% TSR—would be a meaningful stress-test signal for the agent community. However, the central claim depends on temporal alignment between a short reference plan and up-to-30-round evaluated agents, and on the validity of a reference produced and judged by the same model.","major_comments":[{"comment":"The co-temporal ground-truth assumption is not established. Reference plans average 5.4 tool calls (Introduction) while evaluated agents may use up to 30 rounds (§4.1). If both start at the same wall-clock time, the reference typically finishes much earlier; if a live backend changes between the reference's last call and the agent's last call, the judge compares the agent's final result to a stale reference. A correct answer based on the agent's own recent calls can be scored wrong, and a stale answer can be rewarded. The stability runs (Table 3) only vary seeds under the identical protocol and cannot detect this bias; the human study (§4.4) validates the judge against the same reference, not the reference's temporal validity. Because the headline sub-60% claim depends on GPT-5's 58.42% (Table 2), this is load-bearing. The paper should report per-task wall-clock timing and show robustnes","section":"§3.2/§4.1, Fig. 1"},{"comment":"The reference executor and the LLM judge are both GPT-4.1 (§4.1). This couples ground-truth construction with scoring. If GPT-4.1's validated plan or post-processing systematically misses a valid alternative or contains a subtle error, all evaluated agents inherit that bias. The human agreement study (§4.4) uses the same reference, so it establishes judge consistency but not reference correctness. Please add an independent audit of a sample of reference outputs against the task queries, or run a second reference model with disagreement analysis, to rule out reference-model artifacts.","section":"§4.1, §4.4"},{"comment":"The headline 'below 60%' is reported without uncertainty. With 101 tasks, GPT-5's 58.42% (59/101) has a 95% binomial CI of roughly 48–68%, so the point estimate is not statistically distinguishable from 60%. Reporting exact binomial or bootstrap confidence intervals for TSR and for the key model comparisons would support the qualitative claim and prevent over-interpretation of small differences in Table 2.","section":"Abstract; Table 2"}],"minor_comments":[{"comment":"Typo: 'an positive association' should be 'a positive association'.","section":"§4.2"},{"comment":"The Greek kappa symbol appears as a placeholder glyph; ensure it renders as κ.","section":"Fig. 6 caption"},{"comment":"State explicitly whether the stability runs cover all 101 tasks or a subset. Stability is reported for only 6 of the 18 evaluated models; note the limitation.","section":"Table 3"},{"comment":"Clarify whether a fresh reference output is produced once per task or once per evaluated agent/run. This matters for the co-temporality claim.","section":"§4.1"},{"comment":"The human-in-the-loop plan validation is described only briefly. Please specify the number of annotators, the criteria for 'validated', and any inter-annotator agreement, so readers can judge the independence of the ground truth.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and evaluation framework are well designed and the human/stability checks are strong. The main risk is the unverified co-temporal reference assumption and the lack of confidence intervals around the headline TSR. If the authors can supply timing/drift analysis and uncertainty quantification, I would be willing to reconsider. No concerns about novelty or scope; the paper fits CS.CL/agent benchmarking venues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid benchmark paper with a real methodological idea. The parallel-reference evaluation — a GPT-4.1 agent executing a validated plan at roughly the same time as the evaluated agent — is a sensible way to get checkable ground truth on live, time-varying backends. That's new, as far as I know, and it's a meaningful step beyond the static or narrow MCP benchmarks cited. The seven-mode failure taxonomy is also useful and clearly grounded in the logs. The human-judge agreement (kappa above 0.85 on results, 0.78 on trajectories) and cross-judge stability are real evidence that the scoring is consistent, and the three-run stability table is a nice touch.\n\nThe main soft spot is exactly what the stress test flags: temporal co-alignment. Reference plans average 5.4 tool calls; evaluated agents can run up to 30 rounds. On live data, the reference can capture an older state. A correct answer based on fresher data could be scored wrong against a stale reference, and a lucky stale answer could be scored right. This could systematically deflate or inflate TSR. The stability runs and human study do not address it — they validate the judge, not the reference's temporal validity. The paper needs an extra experiment: rerun the reference at different time offsets, or at least report the time gap between reference completion and agent completion, and show results are insensitive. Without that, the headline “even frontier LLMs achieve a success rate below 60%” is provisional.\n\nAlso worth noting: GPT-4.1 serves as both the reference executor and the judge. That's a mild same-model dependency, though the human agreement study mitigates it. And there's no code/data link in the visible text, which is a practical problem for reproducibility. These are fixable.\n\nWho's it for? Anyone building or evaluating MCP agents, and anyone benchmarking agentic tool use in live environments. It deserves a serious referee — the design is thoughtful and the failure analysis is genuinely useful. I'd want the temporal drift question answered before accepting it as is, but this is a conditional accept, not a reject.","headline":"Worthwhile MCP benchmark with a genuinely useful parallel-reference design, but I wouldn't bank on the sub-60% headline until the temporal-alignment issue is addressed.","tokens_in":663,"tokens_out":1557,"would_cite":true,"duration_ms":33985,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LiveMCP-101, a 101-query benchmark of live multi-tool MCP tasks, finds that even the strongest LLM agent succeeds on only 58.42% of tasks and classifies seven failure modes.","keywords":["LiveMCP-101","Model Context Protocol","multi-tool agents","benchmark","parallel evaluation","LLM-as-a-judge","failure analysis","token efficiency"],"falsifier":"Take a query whose live data is known to change mid-evaluation (e.g., a stock price or sports score updated every minute). If the reference agent polls early, and a correct agent later answers with the updated value, the judge must decide whether to reward the updated answer. If such cases are systematically scored wrong, the parallel framework's temporal alignment claim is falsified; measuring the TSR difference between same-window and delayed-reference runs would quantify the drift penalty.","tokens_in":9043,"feed_emoji":"🤖","tokens_out":5301,"duration_ms":49022,"temperature":0.7,"pith_summary":"The paper introduces LiveMCP-101, a benchmark of 101 real-world queries that each require an agent to discover and coordinate multiple tools available through the Model Context Protocol, and claims it is the first to do so against live, time-varying backends while keeping ground truth checkable. To handle the fact that live tool answers change over time, the benchmark runs a reference agent—following a human-validated execution plan—at the same time as the evaluated agent, and scores the evaluated agent against that co-temporal output. Across 18 models, the best result is GPT-5 at 58.42% task success overall and 39.02% on the hardest tier, with every open-source model below 23%. The paper also classifies failures into seven subtypes spanning planning, parameterization, and output handling, which it presents as concrete targets for improving MCP-capable agents.","feed_headline":"Frontier LLM agents succeed on only 58% of live MCP tasks","feed_subtitle":"Runs live MCP tasks with a simultaneous reference agent for scoring; seven failure modes hit planning, parameters, and output handling.","key_machinery":"The parallel evaluation framework is the load-bearing mechanism. A reference agent executes a pre-validated plan in real time, producing a co-temporal reference output while the evaluated agent works autonomously on the same live query; an LLM judge then compares final outputs and trajectories. The validated execution plans (with explicit parameters and post-processing code) make ground truth objective despite live data drift, and the per-task MCP pools (15 servers, 76–125 tools) provide distractors that force genuine tool discovery.","core_discovery":"LiveMCP-101's central claim is that current frontier LLM agents cannot reliably execute realistic multi-step tasks that require dynamically discovering and coordinating multiple MCP tools against live backends. The paper establishes this with a 101-query benchmark across 41 MCP servers and 260 tools, stratified into three difficulty tiers, where each query has a validated plan averaging 5.4 tool calls. Evaluation runs an evaluated agent and a GPT-4.1 reference agent that follows the validated plan simultaneously, then has an LLM judge score the final answer and trajectory against the reference, with human-judge agreement above 0.85. The reported task success rates—58.42% for the strongest mo","pith_inferences":["If the co-temporal premise holds, LiveMCP-101's difficulty ranking should generalize to other live tasks; a natural test is to run the same queries with delayed reference execution and measure TSR drift.","The reference agent's explicit plan may undercut flexibility: agents that find a better route than the validated plan could be scored lower, so the benchmark measures adherence-plus-success rather than pure task completion.","The failure taxonomy suggests concrete training data: injecting MCP-schema examples into post-MCP models could reduce syntactic errors, while instruction tuning on verifying parameters against tool schemas could cut semantic errors.","The per-task pool with distractors models real deployments, but the fixed 15-server pool may penalize models with smaller context windows more than their planning ability warrants; context-length ablations would separate these factors."],"forward_implications":["Benchmarks for live agent evaluation must include co-temporal reference outputs; static or mock environments cannot fairly measure agents against changing data.","The under-60% ceiling for frontier models indicates that multi-step MCP orchestration—not single-call tool use—is a current bottleneck for autonomous agents.","The seven failure subtypes give a taxonomy for targeted improvements: semantic parameter errors and output parsing errors dominate, while syntactic errors are rare except in models trained before MCP adoption.","Token-efficiency trends suggest that for closed models, planning quality drives success more than raw token budget; open models convert tokens into evidence less efficiently.","The ablation results imply increasing agent round limits beyond ~25 yields no gain, so effort should shift from budget to planning quality."],"supporting_citations":[{"why":"Defines the Model Context Protocol that the benchmark evaluates.","marker":"Anthropic, 2024"},{"why":"Provides the ReAct agent framework used for execution.","marker":"Yao et al., 2023"},{"why":"Supplies the LLM-as-a-judge evaluation method.","marker":"Zheng et al., 2023"},{"why":"Identifies GPT-4.1 as the reference agent and judge.","marker":"OpenAI, 2025a"},{"why":"Concurrent live-evaluation benchmark whose lack of verifiable ground truth LiveMCP-101 addresses.","marker":"Mo et al., 2025"},{"why":"Concurrent live evaluation lacking objective verification, motivating the parallel reference design.","marker":"Wang et al., 2025"},{"why":"Provides the quadratic-weighted Cohen's kappa used in human-judge agreement.","marker":"Cohen, 1960"}],"fun_headline_variants":["LLM agents fail 42% of live MCP tasks","Only 58%: LLM agents stumble on live MCP tasks","LiveMCP-101: LLMs score 58% on real MCP tasks","Seven failure modes plague MCP agents in live benchmark","MCP agents: 58% success, 7 failure modes"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim rests on the assumption that a reference answer produced by following a fixed validated plan at roughly the same wall-clock time is a valid gold standard for every evaluated agent, so that live data drift does not reward stale references or penalize correct updated answers.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents fail 42% of live MCP tasks","Only 58%: LLM agents stumble on live MCP tasks","LiveMCP-101: LLMs score 58% on real MCP tasks","Seven failure modes plague MCP agents in live benchmark","MCP agents: 58% success, 7 failure modes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001428,"raw_usage":{"total_tokens":5591,"prompt_tokens":732,"completion_tokens":4859,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":4781}},"tokens_in":476,"tokens_out":4859,"duration_ms":37131,"temperature":1.0,"reasoning_tokens":4781,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:41:32.190981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a query whose live data is known to change mid-evaluation (e.g., a stock price or sports score updated every minute). If the reference agent polls early, and a correct agent later answers with the updated value, the judge must decide whether to reward the updated answer. If such cases are systematically scored wrong, the parallel framework's temporal alignment claim is falsified; measuring the TSR difference between same-window and delayed-reference runs would quantify the drift penalty.","supporting_citations":[],"review_version":1}