{"id":"c3ceedad-ddc2-4f9b-ad0b-5163e05c22be","arxiv_id":"2506.07672","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MCPWorld provides the first MCP-enabled, white-box benchmark for API, GUI, and hybrid computer use agents, with internal-hook verification over 201 tasks.","lead":"MCPWorld is a new testbed that lets researchers compare computer use agents that operate through a graphical interface, through APIs, or through both, using 201 tasks in 10 open-source desktop apps. It verifies task completion by hooking into application internals, and reports that a hybrid agent using Claude 3.7 Sonnet succeeded on 75.12% of tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline modality comparison is confounded by BASH tool availability: main 'MCP-only' and 'Hybrid' runs include BASH, and Appendix D's BASH-disabled numbers show Hybrid falls below GUI-only.","rationale":"The reader's CONDITIONAL verdict is appropriate, but my reason for conditionality differs. The reader identified the unvalidated verification hooks as the weakest assumption; that is a genuine concern for the benchmark's core evaluation validity. However, I find an even more immediately decisive problem in the empirical comparison: the paper's own appendix shows that the reported modality results include BASH, and that the hybrid advantage disappears when BASH is removed. This is not a matter of missing error bars or speculation about hook quality; it is an internal inconsistency between Section 4.1 and Appendix D. Since the abstract and Section 4.3 use the 75.12% hybrid result as 'initial evidence on the practical effectiveness of agent automation leveraging MCP,' the empirical central claim is materially weakened. The benchmark infrastructure contribution can still stand, but the modality comparison needs to be redone with BASH controlled, which is why I recommend keeping the reader's CONDITIONAL verdict. I credit the authors for including the BASH ablation in Appendix D, but the main text does not integrate its implications. My proposed test directly isolates the confound and would settle whether the modality findings survive.","tokens_in":25904,"tokens_out":8669,"duration_ms":109790,"concrete_test":"Rerun the full 201-task suite with the released code in a controlled design that separates BASH from modality: GUI-only with and without BASH, MCP-only with and without BASH, Hybrid with and without BASH, plus a BASH-only condition (no GUI, no MCP), with the same 3 runs per condition. If the only conditions in which Hybrid exceeds GUI-only are BASH-enabled, or if BASH-only performance is comparable to MCP-only, then the modality claims and Table 4 should be revised to report the BASH-free numbers and acknowledge that the observed hybrid advantage is attributable to shell access rather than to MCP/API integration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 claims the MCP-only agent 'could only use these MCP tools, forcing reliance on the programmatic interface,' and the paper's headline numbers (GUI-only 70.65%, MCP-only 53.23%, Hybrid 75.12%) are presented as a clean comparison of interaction modalities. But Appendix C states that in the MCP-only setup 'BashTool and the EditTool might still be available depending on the specific experimental variant,' and Appendix D/Figure 4 reveal that the main-table numbers are the Bash-enabled results. When BASH is disabled, success rates become: GUI-only 67.00%, MCP-only 43.00%, Hybrid 65.50%. The hybrid advantage over GUI-only reverses (65.50% vs 67.00%), and MCP-only drops by 10.23 percentage points. Since the GUI-only configuration also includes BASH, the reported three-way comparison does not isolate API, GUI, or hybrid interaction modalities; it measures combinations of BASH plus these modalities. Consequently, the central empirical claim that 'enabling MCP enhances the capabilities of CUA' and the failure attributions in Table 5 are not supported by the reported experiments. The internal inconsistency between Section 4.1 and Appendix D makes this a load-bearing concern for the empirical layer of the paper, even though the benchmark infrastructure itself may remain useful.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MCPWorld proposes a benchmark testbed for computer-use agents (CUAs) that interact with desktop applications through GUI, API (MCP), or hybrid interfaces. The framework uses ten open-source desktop applications, 201 curated tasks, and a white-box evaluation paradigm that verifies task completion by instrumenting application internals rather than by matching screenshots or output files. The paper reports preliminary experiments with a Claude 3.7 Sonnet-based agent: a hybrid configuration reaches 75.12% task success, compared with 70.65% for GUI-only and 53.23% for MCP-only, and an appendix ablates the effect of removing the BashTool. The central empirical claim is that adding MCP tools enhances CUA capability, with the hybrid configuration outperforming both unimodal baselines.","tokens_in":26092,"tokens_out":3979,"duration_ms":50144,"significance":"If the benchmark is taken as a resource rather than as a clean modality comparison, it is a potentially valuable contribution: it is fully containerized, uses open-source applications, integrates existing MCP servers, and its dynamic instrumentation / code-injection / API-query evaluation is a genuinely different verification approach from UI matching or output-file comparison. Task curation appears careful, with two cross-validations per task, DuckTrack human traces, and detailed per-task verification logic. However, the paper's headline empirical comparison is confounded by the availability of the BashTool, and the reported differences lack uncertainty quantification. The benchmark infrastructure is therefore useful and worth publishing after revision, but the current empirical evidence for the claim that MCP enhances CUA capability is not convincing as reported.","major_comments":[{"comment":"The three-way modality comparison is confounded by BashTool availability. Section 4.1 says that in the MCP-Only setup 'the agent could only use these MCP tools, forcing reliance on the programmatic interface,' but Appendix C states that 'BashTool and the EditTool might still be available depending on the specific experimental variant,' and Appendix D reports that the main-table numbers are the Bash-enabled results. Since GUI-Only also included BashTool, the headline comparison measures 'Bash plus modality' rather than isolated GUI, API, or hybrid interaction. Appendix D shows that when BashTool is disabled, the success rates become GUI-only 67.00%, MCP-only 43.00%, and Hybrid 65.50%: the hybrid advantage over GUI-only reverses, and MCP-only drops substantially. Consequently, the claims that 'enabling MCP enhances the capabilities of CUA' and the failure attributions in Table 5 are not supported by the reported experiments. The authors should either make the no-BASH comparison primary or fully cross BashTool availability with each modality, and should temper the conclusions accordingly.","section":"Section 4.1 and Appendices C/D"},{"comment":"All performance comparisons are reported as point estimates over three attempts per task with no confidence intervals, standard errors, or significance tests. The GUI-only versus Hybrid gap in Task Success Rate is 4.47 percentage points, which is small relative to the likely per-task variance across 201 tasks, and even the larger MCP-only gap is asserted without uncertainty quantification. The paper should report per-task variance, bootstrap confidence intervals, or paired significance tests before drawing conclusions about the relative ordering of the three configurations.","section":"Section 4.2 and Table 4"},{"comment":"The validity of the benchmark's evaluation depends on author-written verification hooks capturing all legitimate completion paths and firing only on semantically correct events. Appendix D explicitly acknowledges that agents can use BashTool to bypass annotated key steps while still achieving final success, which shows that the key-step layer is not a complete model of task progress. The paper should validate hook coverage (for example, against human traces and against multiple solution paths), report hook precision and recall if possible, and discuss how missed or spurious hook firings could affect both Task Success Rate and Key Step Completion Rate.","section":"Section 3.2 and Appendix D"},{"comment":"The failure-reason attribution in Table 5 is load-bearing for the discussion of MCP coverage, but the manuscript does not state how the labels were obtained: whether by LLM judgment, log inspection, human annotation, or an automatic classifier, and whether each failure was labeled once per run or per task. This method must be specified, and ideally validated with inter-annotator agreement or a manual audit, before the table can be used to support the claim that MCP-only underperformance is due mainly to 'Insufficient MCP coverage.'","section":"Section 4.3 and Table 5"}],"minor_comments":[{"comment":"There is a typo in the sentence 'a more flexible action space that provices CUAs more freedom,' which should read 'provides.'","section":"Section 4.3"},{"comment":"The QGIS LOC entry appears as '1.3×10 6' with a malformed superscript; it should read '1.3×10^6' or equivalent.","section":"Table 2"},{"comment":"The text lists three input modes (GUI-only, CLI-only, and multimodal), but the experiments only compare GUI-only, MCP-only, and Hybrid; the relationship between 'CLI-only' and 'MCP-only' should be clarified.","section":"Section 2.2"},{"comment":"The difficulty thresholds '0-5', '5-10', and '10+' have overlapping boundaries at 5 and 10 steps; the paper should specify whether the boundaries are inclusive or exclusive.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best positioned as a testbed/resource contribution rather than as a definitive empirical study of MCP's benefit. The infrastructure, task curation, and white-box evaluation are strong, but the headline modality comparison is confounded by BashTool availability and the reversal in Appendix D is serious enough that the empirical claims need to be reworked. The issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MCPWorld is worth taking seriously. The white-box internal-hook evaluation is genuinely new, and the 201-task, 10-app, MCP-enabled testbed is a real resource for the computer-use-agent field. The authors put real work into task curation—cross-validation, DuckTrack human traces, and detailed verification logic—and they ship the code and data. That alone makes this a useful paper.\n\nThe soft spots are real but concentrated in the empirical layer. The headline three-way comparison (GUI 70.65%, MCP 53.23%, Hybrid 75.12%) is presented as a clean modality comparison, but main-table numbers are BASH-enabled. Their own Appendix D shows that when BASH is disabled, Hybrid drops to 65.50% and GUI-only to 67.00%—the hybrid advantage reverses. MCP-only falls to 43.00%. So Section 4.2's claim that 'enabling MCP enhances the capabilities of CUA' is not actually supported by the reported experiments. The paper is honest enough to include the ablation, but the main text doesn't carry the caveat forward. That needs fixing.\n\nRelated issues: no confidence intervals across the three attempts per task; failure attribution in Table 5 comes from an unspecified method; and the verification hooks are treated as ground truth without independent validation against human judgment. The authors do note in Appendix D that agents can bypass annotated key steps via BASH while still succeeding, which is a fair admission but also shows the key-step metric isn't a complete model.\n\nNone of this sinks the benchmark. The infrastructure, the hooking evaluator, and the task set are useful regardless of how the first agent experiments come out. MCP server coverage varies because the MCP servers are community projects, so the MCP-only numbers are partly measuring server maturity—worth stating explicitly, but not a fatal flaw for a benchmark paper.\n\nWho should read it: anyone building or evaluating computer use agents, particularly people working on MCP and hybrid interaction. It deserves a serious referee: the contribution is real, and the empirical overreach is correctable with a re-analysis that controls BASH and reports uncertainty. I'd accept it for review with major revisions.","headline":"MCPWorld is a genuinely useful white-box benchmark for GUI/API/hybrid agents; the headline modality comparison is real but confounded by BASH availability, so read the empirical claims with caution.","tokens_in":26709,"tokens_out":2129,"would_cite":true,"duration_ms":25691,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MCPWorld proposes that computer-use agents be evaluated by hooking internal application behavior, and reports that a hybrid GUI-plus-API agent reaches 75.12% task success across 201 desktop tasks.","keywords":["computer use agents","benchmarking testbed","Model Context Protocol (MCP)","white-box evaluation","GUI automation","API agents","hybrid agents","desktop applications"],"falsifier":"Take any MCPWorld task, complete it by directly editing the underlying file or database that the task is meant to change, and see whether the hook-based evaluator marks it successful; if it does not, the hooks have missed a legitimate completion path. A second check would have human judges score a random sample of recorded runs and compare their verdicts with the hook-based scores, looking for systematic disagreement on tasks without a unique hooked path.","tokens_in":25655,"feed_emoji":"🖥️","tokens_out":7893,"duration_ms":83700,"temperature":0.7,"pith_summary":"MCPWorld sets out to make computer-use agents comparable across every way they can interact with a computer: through a graphical interface, through application APIs, or through both. Its central proposal is to evaluate agents with 'white-box apps'—open-source applications that can be recompiled and instrumented so the evaluator verifies task completion by hooking internal functions and events instead of matching screenshots, UI trees, or output files. The paper argues this decouples evaluation from the agent's implementation and from UI state, allowing one testbed to grade GUI-only, MCP-only, and hybrid agents on the same 201 tasks. A representative LLM-powered agent reaches 75.12% task success in hybrid mode, above GUI-only (70.65%) and MCP-only (53.23%), and the authors read this as initial evidence that combining MCP APIs with GUI fallbacks helps agents on complex desktop tasks. If the paradigm is sound, agent benchmarks can grade what the application actually did rather than what the screen looked like.","feed_headline":"Hybrid desktop agents beat GUI-only by five points on 201 tasks","feed_subtitle":"Combining GUI tools with MCP APIs reaches 75.12% success; using either alone falls to 70.65% and 53.23%.","key_machinery":"The load-bearing mechanism is the white-box application and its verification hooks. Each of the ten applications is open source, so MCPWorld can recompile it, attach MCP servers, and place evaluation logic at semantically meaningful internal events—function calls, event emissions, or state transitions—using dynamic instrumentation, targeted code injection, or API-driven state queries. When the agent acts, the hooks fire at the exact moment the action takes effect inside the app, letting the evaluator verify both intermediate key steps and final completion without querying the screen, matching a trajectory, or waiting for a file to be written. Around this, a unified tool space presents the agent with GUI tools (screenshots, mouse and keyboard) and MCP tools (application-specific functions) so the same task can be attempted by GUI, API, or hybrid policies and judged by the same internal signals.","core_discovery":"The paper's central claim is that computer-use agents are best evaluated from inside the application, not from outside it. It proposes that task completion be verified by white-box hooks that observe internal function invocations, events, and state transitions, which it argues are more accurate, more robust to UI changes, and more fair across interaction modes than screenshot matching, UI-tree matching, or output-file matching. To demonstrate, it builds MCPWorld, a containerized desktop testbed with 201 tasks across 10 open-source applications, where each task has annotated key steps and hook-based success criteria. Running a representative LLM-powered agent on all 201 tasks, the paper reports 75.12% task success for a hybrid agent with both GUI and MCP tools, 70.65% for GUI-only, and 53.23% for MCP-only, with the MCP-only shortfall attributed mainly to limited MCP coverage and tool descriptions rather than to planning ability; on hard tasks, the hybrid's smaller drop in success is taken as evidence that API access adds robustness.","pith_inferences":["Editorial inference: the same hook-based verification idea could move from benchmarks into production settings, where an application's internal events could serve as ground truth for whether an agent's action actually took effect.","Editorial inference: because the MCP-only score depends on the maturity of community-written MCP servers, the 53.23% figure measures today's MCP ecosystem as much as agent capability; a better server for one application could shift the comparison.","Editorial inference: the paper's observation that hybrid success dips on medium tasks suggests tool-call prompt length and choice overhead can offset API benefits, a variable future benchmark designers could control explicitly by compressing tool descriptions.","Editorial inference: hook placement is per-task annotation work, so scaling to more apps will depend on whether LLM-assisted hook discovery, which the paper reports using, can be automated without losing verification fidelity."],"forward_implications":["Evaluator scores become robust to cosmetic UI changes, because success is determined by internal state transitions rather than pixels, accessibility trees, or output files.","The same app instance can fairly compare GUI-only, API-only, and hybrid agents, and the paper's three-way success rates demonstrate that comparison on a shared testbed.","MCP access helps most on complex tasks: hybrid success falls 39.3 points from easy to hard, versus 54.9 for GUI-only, so API tools provide a fallback that preserves partial progress.","Tasks whose completion exists only in memory, such as a breakpoint hit with its call stack, become verifiable even when nothing is ever written to disk.","The config-driven design means the suite can grow: adding a new app or task requires registering verification logic and state snapshots rather than building a new evaluator from scratch."],"supporting_citations":[{"why":"Defines the Model Context Protocol that MCPWorld adopts as the standardized API tool interface.","marker":"[2]"},{"why":"A desktop GUI benchmark whose external UI matching and output-file matching are the baselines MCPWorld argues against.","marker":"[20]"},{"why":"The only prior testbed supporting both GUI and API agents, which MCPWorld extends by adding MCP and app-internal hooking.","marker":"[26]"},{"why":"Supplies the computer-use agent demo that the paper adapts as its reference agent implementation.","marker":"[1]"},{"why":"A mobile benchmark using external state matching, cited as representative of the evaluation paradigm MCPWorld replaces.","marker":"[17]"},{"why":"Provides the dynamic instrumentation mechanism used to hook compiled applications at runtime.","marker":"[15]"},{"why":"The ReAct-style reasoning-and-acting strategy the reference agent uses for planning and tool execution.","marker":"[24]"}],"fun_headline_variants":["White-box hooks beat screenshots for agent eval","Hybrid agent tops 75% on 201-task MCPWorld suite","API-GUI combo lifts agent success 5 points","MCPWorld testbed scores agents from the inside","GUI+API agent hits 75% on MCPWorld benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark assumes that the hooks placed in the application cover every legitimate way to finish a task, yet the paper's own appendix shows agents can bypass annotated key steps through shell commands while still succeeding.","fun_headline_variants_meta":{"raw":{"variants":["White-box hooks beat screenshots for agent eval","Hybrid agent tops 75% on 201-task MCPWorld suite","API-GUI combo lifts agent success 5 points","MCPWorld testbed scores agents from the inside","GUI+API agent hits 75% on MCPWorld benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1449,"prompt_tokens":1098,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":268}},"tokens_in":714,"tokens_out":351,"duration_ms":4749,"temperature":1.0,"reasoning_tokens":268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:28:35.653698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any MCPWorld task, complete it by directly editing the underlying file or database that the task is meant to change, and see whether the hook-based evaluator marks it successful; if it does not, the hooks have missed a legitimate completion path. A second check would have human judges score a random sample of recorded runs and compare their verdicts with the hook-based scores, looking for systematic disagreement on tasks without a unique hooked path.","supporting_citations":[{"cited_title":"Introduction: Get started with the model context protocol (mcp)","cited_arxiv_id":null,"evidence_quote":"Defines the Model Context Protocol that MCPWorld adopts as the standardized API tool interface."},{"cited_title":"Agentstudio: A toolkit for building general virtual agents, 2025","cited_arxiv_id":null,"evidence_quote":"The only prior testbed supporting both GUI and API agents, which MCPWorld extends by adding MCP and app-internal hooking."},{"cited_title":"Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku","cited_arxiv_id":null,"evidence_quote":"Supplies the computer-use agent demo that the paper adapts as its reference agent implementation."},{"cited_title":"frida: Dynamic instrumentation toolkit for developers, reverse-engineers, and security researchers.https://frida.re/","cited_arxiv_id":null,"evidence_quote":"Provides the dynamic instrumentation mechanism used to hook compiled applications at runtime."}],"review_version":1}