{"id":"d77fddf8-dcd4-4ddf-ac85-7f7505880e77","arxiv_id":"2608.00150","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A large-scale dynamic audit found most public MCP servers have no authentication, dozens have exploitable flaws, and nearly half disappear within three days.","lead":"This paper scanned the public internet for servers that let AI assistants use external tools, then actively probed hundreds of them for security holes. It found that most have no login, that many expose dangerous commands without checks, and that large numbers vanish within days.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline vulnerability counts rest on unvalidated heuristic detectors—SSRF threshold calibrated in-sample, static tool-name inference for shell execution—so the 68-GHSA and 687-tool claims are not yet established.","rationale":"The reader's weakest-assumption analysis correctly identifies the unvalidated mapping from heuristic signals to vulnerability classes as the load-bearing risk. The paper is transparent about this in Section VIII, but the abstract and central claim present the 68 GHSAs and category prevalence as established findings, without the precision/recall caveat. Manual triage by the author mitigates but does not eliminate this risk, especially since 49 of 68 advisories remain under embargo and cannot be independently inspected from the preprint. The SSRF threshold being calibrated on the very case used as confirmation is a concrete instance of the problem. I also note an additional internal inconsistency: the claim of findings in all MST-10 categories is contradicted by Table V's omission of MCP02 and MCP09, though this is secondary to the validation concern. Because the broad picture—unauthenticated MCP deployments, real exploitable flaws, high churn—retains support even if the exact counts shift, CONDITIONAL remains the appropriate verdict; my stress-test does not move it, so UNCHANGED is recommended.","tokens_in":17039,"tokens_out":4118,"duration_ms":49867,"concrete_test":"Build a labeled benchmark of roughly 50 MCP servers—20 with known ground-truth vulnerabilities (e.g., SSRF, command injection, schema bypass), 20 benign, and 10 patched versions—deployed in a sandboxed environment. Run Corvus v1.3.1 modules blind at the confidence≥70 threshold and compute per-module precision and recall. If SSRF precision falls below 0.9 or command-injection/shell-tool precision falls below 0.8, the 68 and 687 headline counts must be revised downward; this also directly tests whether the >5s SSRF threshold generalizes beyond the epwforge in-sample calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's quantitative spine is the 68 reportable GHSAs and the per-category prevalence in Table V, but those numbers inherit the accuracy of Corvus's heuristic detectors, for which no ground-truth precision/recall is reported. Section IV.C.3.d defines SSRF confirmation as a >5-second response-time delta and says the threshold was \"calibrated empirically\" on the same epwforge case later presented as confirmation in Section V.D—an in-sample calibration that cannot independently validate the detector. Similarly, the 687 unauthenticated shell-execution tool instances are inferred from tool names and schemas matching exec-like patterns (Section V.B), not from observed command execution, and the count spans the \"broader confirmed set\" rather than only the 414 dynamically audited servers. Section VIII explicitly concedes that module-level precision and recall \"is ongoing work and is not reported here.\" Without a labeled benchmark, false positives in any of these detectors directly inflate the 68-GHSA total and the severity distribution. A secondary internal inconsistency: the central claim asserts findings \"spanning all MCP Security Top 10 (MST-10) categories,\" but Table V contains no MCP02 or MCP09 rows, and MCP09 was explicitly excluded per ethical constraints (§VII).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the first dynamic behavioral security assessment of internet-facing Model Context Protocol (MCP) servers. The authors combine passive discovery across eleven sources (certificate-transparency logs, package registries, code hosts, Internet scanners, and MCP directories) with active testing by an open-source 34-module framework, 'Corvus,' across four measurement runs in July 2026. They confirm 640 unique production MCP servers, dynamically audit 414, and report: 91.8% (380/414) lacking OAuth; 687 tool instances across the confirmed pool advertising unauthenticated shell-execution capabilities (identified via tool name/schema inspection); 68 reportable vulnerabilities filed as GitHub Security Advisories (19 public, 49 embargoed), including SQL injection, SSRF against cloud metadata services, prompt template injection, and cursor-based path traversal; and 41.6% churn of confirmed servers between the July 21 and July 24 runs. The paper introduces the MST-10 taxonomy of MCP-specific vulnerability classes, characterizes deployment churn and the near-total authentication gap, documents a responsible-disclosure pipeline, and releases its tooling. The central conclusion is that the MCP deployment surface is large, rapidly changing, largely unauthenticated, and vulnerable across the introduced taxonomy.","tokens_in":17295,"tokens_out":17207,"duration_ms":176681,"significance":"If the headline measurements hold, this is a substantial contribution: it is the first behavioral, deployment-level security measurement of the MCP ecosystem, with reproducible open-source tooling (Corvus, Petrel), a multi-source discovery methodology, SARIF output, internally consistent arithmetic (380/414 = 91.8%; 193/464 = 41.6%; 972 observations minus 332 duplicates = 640 unique), and a carefully documented 90-day disclosure pipeline with 19 public advisories. The near-universal OAuth absence and the extreme churn are directly measured, credible results that stand on their own as an empirical baseline. The paper is also commendably transparent: it concedes the unreported module-level precision/recall, the heuristic nature of confidence thresholds, the single-researcher confirmation-bias risk, and the temporal validity limits. The weakness is that the quantitative spine of the paper — the 68-GHSA total, the per-category prevalence in Table V, and the 687 shell-execution figure — rests on heuristic detectors whose false-positive rates are unmeasured and, in the SSRF case, calibrated on the very observation used as confirmation. These are fixable within scope, but as written the s","major_comments":[{"comment":"The headline quantitative claims — 68 reportable GHSAs, per-category prevalence in Table V, severity distribution — inherit the accuracy of Corvus's heuristic detectors (timing deltas, response-size differentials, schema-name inspection). Section VIII explicitly concedes that module-level precision/recall against a labeled benchmark 'is ongoing work and is not reported here,' and §IV.C.4 calls the confidence thresholds 'empirically calibrated heuristics.' Manual triage (§IV.D) mitigates, but manual confirmation uses the same behavioral signals. Table V's finding-server pairs sum to 655 while only 68 advisories were filed; the mapping from module findings to GHSA filings is never stated, so the reader cannot tell which rows are raw SARIF outputs versus post-triage confirmed findings. I request out-of-sample calibration on the 19 disclosed advisories plus confirmed negatives, and an explic","section":"§VIII; §IV.C.4; Table V"},{"comment":"The SSRF threshold is calibrated in-sample. §IV.C.3.d defines SSRF confirmation as a response-time delta exceeding 5 s and states that 'this threshold was calibrated empirically; we observed a 11.9 second response versus a 0.3 second baseline in one confirmed SSRF case,' and that same case is then presented in Case Study 2 as the confirmation of SSRF in epwforge. The threshold therefore has no out-of-sample validation, and all 29 SSRF findings in Table V inherit it. The epwforge case itself is plausible, but confirmation is a pure timing oracle — §VII states no IMDS content was retrieved — so a slow network path or rate limiting cannot be excluded. Fixing the threshold on a held-out set of known-SSRF and known-clean servers before evaluation, or reporting sensitivity of the 29-finding count to the threshold, would establish this detector.","section":"§IV.C.3.d; §V.D, Case Study 2"},{"comment":"The central claim in §V.E that findings span 'all MCP Security Top 10 (MST-10) categories' is internally inconsistent with Table V, which has no MCP02 row, and with §IV.A/Table V, which state MCP09 (Denial of Service) was excluded from execution per ethical constraints. At most eight of ten categories have reported findings. The gap is compounded by the taxonomy definition: MCP02 is 'unauthenticated tools/call invocation reaching privileged operations,' which is exactly what the 687 unauthenticated shell-execution tools (§V.B) represent; zero MCP02 findings alongside 380 MCP08 findings suggests overlapping category boundaries rather than a genuine absence. Please either report MCP02 findings or reclassify the shell-execution exposure, and amend the claim to 'eight of ten categories.'","section":"§V.E; Table V; §IV.A"},{"comment":"The 687 tool-instance figure is static inference, not behavioral confirmation. §V.B identifies shell-execution capabilities from tool 'names and descriptions matching bash_execute, run_command, exec,' and §VII states capability existence was confirmed 'by observing that tool schemas exposed parameters passed to shell invocation functions,' with no commands beyond benign echo-type probes submitted. The assertion that an adversary 'can issue arbitrary operating system commands through the standard MCP tools/call interface' extrapolates from schema inspection to guaranteed RCE; false positives from name-matching (e.g., a safely parameterized subprocess wrapper or a logging tool named 'executeQuery') directly inflate the 687 count. Also, the count draws on the 640-server confirmed pool while the 91.8% OAuth figure is over the 414 audited servers; the abstract's juxtaposition conflates denomi","section":"§V.B; §VII"}],"minor_comments":[{"comment":"Contribution C4 states the 9 absent CRITICAL-rated servers were 'remediated or decommissioned,' but §V.E says 'endpoint disappearance alone cannot confirm remediation' and Figure 3 labels them 'patched / removed' as fact. The measurement supports only 'absent'; the remediation attribution needs a source (e.g., GHSA maintainer responses), which is not provided.","section":"§I (C4); §V.E; Fig. 3"},{"comment":"The MCP09 exclusion is stated in §IV.A, but Table V's caption refers to '§VII' for the exclusion; §VII contains no explicit MCP09-exclusion statement (it says no DoS modules were executed). Align the cross-reference.","section":"§IV.A; Table V caption; §VII"},{"comment":"The confidence-score bands (85–100, 71–84, 40–70, 1–39) are given only in prose; a small table or an explicit mapping from signal type to score would aid replication. Also clarify why lexical-pattern-only findings (1–39) never appear in SARIF while schema-based shell-execution inference is treated as a confirmed capability (§V.B).","section":"§IV.C.4"},{"comment":"MST-10 is described as 'derived from iterative analysis of findings across our measurement campaign' and then applied to classify those same findings. This is a mild circularity worth acknowledging explicitly in §VIII: the prevalence numbers are not independent of how the taxonomy's category boundaries were drawn.","section":"§III.D"},{"comment":"Typos: 'anamplifier' (§I); 'OW ASP' for OWASP (§III.D and references [13], [14]); 'Y . Zhou' and 'Y . Zhao' (references [19], [4]).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"A single-author, independent-researcher manuscript with a well-documented, transparent methodology and a genuinely useful open-source toolchain; the disclosure pipeline is careful and the self-identified limitations are honest. The editor should weigh whether the evidentiary base matches the headline claims: the 19 public advisories are the only independently checkable artifacts yet they are not linked or summarized with reproduction evidence in the paper, and the 49 embargoed advisories cannot be inspected by reviewers. The in-sample SSRF threshold (§IV.C.3.d) and the unreported module precision/recall (§VIII) are the decisive weaknesses; if the authors can provide out-of-sample calibration or materially scale back the prevalence claims, the paper would be a solid empirical contribution. Also note the manuscript is closer in form to a top-tier security-measurement conference paper than a journal article — the analysis depth (no statistical treatment of discovery-source biases, no sensitivity analysis on thresholds) is below what I would expect for a journal-length treatment — but this is a fit/scope judgment for the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First dynamic read of the MCP surface, and it's genuinely novel—prior work stopped at auth headers or static analysis. The auth finding (91.8% no OAuth on 414 audited servers) and the 41.6% three-day churn are concrete, reproducible measurements that should hold up. The eleven-source discovery and the open-source Corvus/Petrel tooling are a real contribution, and the disclosure pipeline is thought through.\n\nThat said, the headlined numbers that will get the press—68 GHSAs, 687 unauthenticated shell tools, SSRF to cloud metadata—rest on heuristic detection with no ground-truth calibration. The SSRF threshold in IV.C.3.d (a >5s response-time delta) was calibrated on the same epwforge case later presented as confirmation in V.D, so it doesn't independently validate. The 687 shell tool instances come from matching tool names/schemas to exec-like patterns, not from ever running a command; the paper says so almost in passing. And the central claim that findings span all ten MST-10 categories is contradicted by the paper's own Table V, which shows no MCP02 row and excludes MCP09 by design. Those are real soft spots, but they're fixable: soften the categorical claim, label the counts as provisional, and ideally release a small labeled benchmark with precision/recall.\n\nThe selection bias in the 414 audited servers is acknowledged but not bounded—the excluded servers could be safer or more dangerous, and the paper doesn't speculate. The single-researcher setup is a limitation, though the released code does make replication possible.\n\nOn balance the central picture—an unauthenticated, high-churn MCP deployment base with genuinely exploitable flaws—is convincing. The exact prevalence numbers are not. The paper is unusually honest about this in Section VIII; the mismatch is between the abstract's strong wording and the evidence in the body.\n\nI'd send this to a serious referee. The methodology and the empirical gap are substantial enough to deserve a careful read; the referee should push for calibration data or revised wording in the claims. I'd also bring it to reading group, because it's a good lesson in how heuristic proxies can shape a headline.","headline":"First dynamic read of the MCP surface, genuinely novel, but the headlined vulnerability counts rest on unvalidated heuristics and need softening before the numbers can be trusted.","tokens_in":17785,"tokens_out":3698,"would_cite":true,"duration_ms":37071,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports the first dynamic security audit of internet-facing Model Context Protocol (MCP) servers, finding 68 exploitable vulnerabilities across 414 audited servers, with 91.8% lacking OAuth authentication.","keywords":["Model Context Protocol","MCP security","dynamic testing","internet measurement","vulnerability disclosure","SSRF","authentication","AI agents"],"falsifier":"A labeled benchmark corpus of MCP servers with known ground-truth vulnerability status, run through Corvus, would measure module-level precision and recall. Specifically, if the SSRF timing threshold (>5s) yields many positives on servers known to lack SSRF, or if the SQL response-size differential misclassifies benign large responses, the vulnerability counts are overstated. Independent manual review of the source code of the 68 reported servers would also confirm or refute exploitability.","tokens_in":16875,"feed_emoji":"🛡️","tokens_out":8616,"duration_ms":76504,"temperature":0.7,"pith_summary":"The paper sets out to establish that the public internet's Model Context Protocol (MCP) server base is both large and dangerously under-secured, and that this can be demonstrated by actively probing live servers rather than only inspecting source code or registry listings. Over four measurement runs in July 2026, the author discovered 640 confirmed MCP servers from eleven passive sources, dynamically tested 414 of them with a custom 34-module framework, and filed 68 coordinated vulnerability advisories, including SQL injection, SSRF to cloud metadata services, and prompt template injection. The headline numbers are 91.8% of audited servers lacking OAuth, 687 tool instances exposing shell-execution capabilities without access control, and 41.6% of confirmed servers vanishing within three days between runs. If these measurements hold, MCP—the emerging integration layer for AI agents—is being deployed at scale with security defaults closer to a weekend experiment than a hardened network service.","feed_headline":"91.8% of internet MCP servers skip authentication","feed_subtitle":"First dynamic audit of 414 MCP servers finds 68 vulnerabilities and shell tools without authentication.","key_machinery":"The load-bearing mechanism is Corvus, an open-source dynamic testing framework contributed by the paper, comprising 34 test modules (13 static, 21 dynamic) mapped to the ten MST-10 vulnerability categories. Corvus confirms vulnerabilities through heuristic behavioral signals rather than source-code inspection: response-size differentials for SQL injection, timing oracles (e.g., >5-second latency delta) for SSRF against cloud metadata endpoints, schema inspection for shell-execution tool exposure, and lexical pattern matching for prompt injection. A companion discovery pipeline (Petrel) aggregates eleven passive sources and performs HTTP-level MCP handshakes to confirm live servers. Together","core_discovery":"The paper's central claim is that the MCP deployment surface is (i) large and rapidly expanding, (ii) predominantly functionally unauthenticated, (iii) host to exploitable vulnerability classes spanning all ten MCP Security Top 10 (MST-10) categories, and (iv) governed by deployment patterns that complicate defensive monitoring and measurement. This is supported by what the author calls the first dynamic behavioral security assessment of internet-facing MCP servers: 414 servers actively audited, 68 reportable vulnerabilities, 91.8% OAuth absence, 687 unauthenticated shell-execution tool instances, and a 41.6% three-day churn rate between consecutive runs.","pith_inferences":["If the heuristic thresholds (e.g., the 5-second SSRF timing delta or the SQL response-size differential) have high false-positive rates, the 68-vulnerability total and per-category counts may overstate actual risk; a labeled benchmark evaluation of Corvus would settle this. This extension is not in the paper but follows directly from its acknowledged calibration gap.","The 687 shell-execution tool count is based on schema-level names and descriptions; the actual number of truly exploitable, network-reachable shell invocations may be lower once per-tool argument validation and sandboxing are considered. This is an editorial caution, not a paper claim.","Because the most persistent servers also carried the highest CRITICAL counts, the churn statistics may understate long-lived risk: an attacker can focus on stable, high-risk endpoints rather than ephemeral ones. The paper hints at this persistence pattern but does not develop a risk-weighted longitudinal model.","If MCP's steward adds mandatory OAuth by default, this measurement can serve as a before/after baseline; a replicable version of the current scan after such a change would directly quantify policy impact. The paper does not propose this experiment."],"forward_implications":["If the reported rates are accurate, the MCP ecosystem currently offers a wide, unauthenticated attack surface for anyone who can reach the public internet, including remote code execution via shell-capable tools.","The protocol and its SDKs need authentication (OAuth 2.1) as a default for HTTP transport, not an opt-in, otherwise the structural gap will persist as adoption grows.","The high churn means vulnerability scanners and responsible-disclosure workflows must treat server identity as unstable; a disappearing endpoint cannot be assumed patched.","Dynamic behavioral testing of live servers is essential for MCP security assessment; static analysis of registries or source code would miss runtime failures such as SSRF and schema bypass.","The 68 coordinated advisories provide the first concrete baseline of MCP-specific vulnerabilities for maintainers and for future regression measurement."],"fun_headline_variants":["MCP audit: 91.8% lack OAuth, 68 vulns","First MCP security audit: 68 vulnerabilities, shell tools exposed","MCP servers: 91.8% unauthenticated, 41.6% vanish in 3 days","687 unauthenticated shell tools on MCP servers","91.8% of MCP servers skip auth, audit finds 68 vulns"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The validity of every prevalence and severity statistic depends on an unvalidated mapping from heuristic signals to vulnerability classes—a >5-second response-time delta is treated as SSRF, a larger response to a SQL payload as SQL injection, and exec-like tool names as unauthenticated shell execution—and the paper reports that module-level precision/recall is ongoing work, so high false-positive rates in these proxies would inflate the 68-vulnerability total and the authenti","fun_headline_variants_meta":{"raw":{"variants":["MCP audit: 91.8% lack OAuth, 68 vulns","First MCP security audit: 68 vulnerabilities, shell tools exposed","MCP servers: 91.8% unauthenticated, 41.6% vanish in 3 days","687 unauthenticated shell tools on MCP servers","91.8% of MCP servers skip auth, audit finds 68 vulns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001191,"raw_usage":{"total_tokens":4758,"prompt_tokens":759,"completion_tokens":3999,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":3893}},"tokens_in":503,"tokens_out":3999,"duration_ms":24897,"temperature":1.0,"reasoning_tokens":3893,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:08:56.802082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A labeled benchmark corpus of MCP servers with known ground-truth vulnerability status, run through Corvus, would measure module-level precision and recall. Specifically, if the SSRF timing threshold (>5s) yields many positives on servers known to lack SSRF, or if the SQL response-size differential misclassifies benign large responses, the vulnerability counts are overstated. Independent manual review of the source code of the 68 reported servers would also confirm or refute exploitability.","supporting_citations":[],"review_version":1}