{"id":"ed879155-20ac-435a-b68f-da080074499e","arxiv_id":"2607.11086","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MCP security scanners flag almost all runtime MCP servers as risky, yet manual and CVE validation show those alerts are inconsistent and often wrong.","lead":"Researchers built MCPZoo, a 64k-server runtime dataset of Model Context Protocol tools, and used it to test eight popular MCP security scanners. Scanner alerts look alarming (96.89% of servers flagged) but are mostly unreliable: low precision, low agreement, and weak recall on known CVEs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Precision sample of 100 may not support the ecosystem-level unreliability claim once high-volume scanners dominate the alert pool.","rationale":"The reader's weakest_assumption correctly flags limited ground-truth scale and the TP rule, but the more load-bearing gap is the mismatch between how the sample is stratified (to avoid high-volume dominance) and how the headline risk rate is defined (union over all scanners). That mismatch is concrete, checkable from the existing dual-reviewer protocol, and directly affects whether 45.53% can be read as evidence that the 96.89% figure is unreliable. Deployment bias and incomplete public artifacts remain real but secondary for the scanner-reliability claim. The multi-scanner inconsistency (Jaccard 15.66%) and low CVE recall already support unreliability independently of the precision number; the concrete test only tightens the precision half of the argument. Hence the verdict stays CONDITIONAL rather than moving to REJECT or ACCEPT, and agreement with the reader is partial: same section and same spirit, different primary soft spot.","tokens_in":28157,"tokens_out":669,"duration_ms":6019,"concrete_test":"Recompute precision on the same dual-reviewer protocol but with a volume-weighted sample: draw ~200 alerts proportional to each scanner's share of the 96.89% union (or sample 100 servers from the union with probability proportional to number of scanners that flagged them), re-label under the same TP rule, and report both unweighted and volume-weighted precision. If volume-weighted precision rises above ~70% while unweighted stays ~45%, the ecosystem-level unreliability claim needs rephrasing; if it stays near or below 45%, the claim is reinforced.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (scanners are not reliable enough for ecosystem-level security claims) rests on average precision of 45.53% from a stratified sample of 100 servers flagged by at least one scanner (Section 5.3, Table 7). The sample is stratified by scanner and risk category so high-volume scanners do not dominate the review, yet the ecosystem-level statistic that 96.89% of 37,288 servers are risky is driven by the union of all scanners, and high-volume low-precision scanners (A.I.G dynamic 10.40%, Agent-Scan 28.21%, mcp-armor 20.34%) contribute the bulk of that union. Because the paper reports unweighted average precision over sampled alerts rather than a volume-weighted or server-level precision on the union, it is possible that the effective precision of the headline 96.89% figure is substantially lower (or, less likely, higher) than 45.53%. The true-positive rule (concrete reachable unsafe behavior) is sound, but the sampling design does not directly estimate the precision of the quantity the paper uses to motivate the claim. The 10-CVE / 38-server recall set is too small and skewed toward command-injection / credential issues to compensate. If volume-weighted precision of the union is still low, the claim holds; if not, the generalization from sample to ecosystem claim weakens.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper constructs MCPZoo, a large runtime-enabled corpus of MCP servers obtained by multi-source collection and a multi-agent Generation–Verification–Diagnosis pipeline that produces Dockerized deployments validated by real MCP protocol handshakes and tools/list. From 156,842 raw market entries it retains 64,611 unique servers, of which 37,288 are interactable (57.7% agent success). Using this corpus it characterizes ecosystem structure (cross-market overlap, 28.3% code-level duplication, weak native deployability, long-tailed tool exposure with 37.66% high-capability tools) and evaluates eight public MCP security scanners under fixed configs. Scanners flag 96.89% of interactable servers as risky by union, yet a dual-reviewed stratified sample of 100 flagged servers yields only 45.53% average precision, pairwise Jaccard agreement is 15.66%, and recall on a 10-CVE / 38-server ground-truth set is 24.17%. The authors conclude that current scanners are not reliable enough for ecosystem-level security claims and release a public query interface over normalized reports.","tokens_in":28586,"tokens_out":854,"duration_ms":6938,"significance":"If the reliability result holds, the paper substantially revises how the community should interpret MCP scanner outputs and provides the first large, protocol-validated runtime substrate for MCP security measurement. Strengths include operational success criteria (JSON-RPC initialize + tools/list), dual-reviewer human validation of deployments (κ=0.95) and precision labels (κ=0.87), controlled multi-scanner execution with a fixed local LLM backend, and an external CVE-mapped recall set rather than circular self-scoring. The public query interface and the scale jump relative to prior dynamic studies (Table 1) are concrete contributions that other groups can reuse. The work is timely for agent-tool security and directly actionable for scanner developers and MCP market operators.","major_comments":[{"comment":"§5.3 / Table 7: The headline claim that scanners are unreliable for ecosystem-level claims is motivated by the 96.89% union risk rate (§5.2, Table 6) but supported by an unweighted average precision of 45.53% over stratified sampled alerts. High-volume low-precision scanners (A.I.G dynamic 10.40%, Agent-Scan 28.21%, mcp-armor 20.34%) dominate the union. The manuscript should report (i) volume-weighted or server-level precision of the union (or of majority-vote / multi-scanner agreement strata) and (ii) confidence intervals or bootstrap estimates for the 100-server sample, so that the precision of the quantity used in the abstract is estimated rather than only per-scanner averages.","section":null},{"comment":"§5.3 / Appendix D (Table 10): The CVE ground-truth set comprises only 10 CVEs affecting 38 servers and is heavily skewed toward command injection and credential leakage. Overall recall of 24.17% is therefore informative but thin for the strong claim of limited recall on confirmed issues. Either expand the set (additional NVD/GitHub-mapped MCP CVEs or a small hand-crafted exploit suite with reachable paths) or explicitly bound the recall claim to the covered vulnerability types and avoid treating 24.17% as a general ecosystem recall figure.","section":null},{"comment":"§6.2 Deployment Bias and §4.2: 42.3% of unique servers remain non-interactable; failures are dominated by external credentials and infrastructure coupling (63.6%). Dynamic scanners and the 96.89% figure are defined only on the deployable subset. The paper should quantify whether non-deployable servers differ systematically in tool capability, market origin, or static scanner flags (MCPScan / A.I.G static can still run on source), and state how this selection may bias the reliability conclusion toward easier-to-sandbox projects.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this is the first work that both (a) gets tens of thousands of real MCP servers into a protocol-validated runtime state and (b) runs eight scanners on that set and checks whether the alerts mean anything. Prior dynamic MCP work tops out around 1,360 servers; they get 37,288 unique interactable ones. That scale is real, not marketing.\n\nWhat is new is the multi-agent deploy/repair/verify loop (Generation, Verification, Diagnosis) that turns wild repos into Dockerized services checked with a real JSON-RPC handshake and tools/list, plus the first ecosystem-scale precision/recall/Jaccard audit of MCP scanners. Ecosystem findings are useful on their own: heavy cross-market overlap, 28% code-level duplication, 78.6% missing Dockerfiles, native success only 19.6% even with full artifacts, and 37.6% high-capability tools. Scanner side: 96.89% of interactable servers flagged by at least one tool, average sampled precision 45.53%, pairwise Jaccard 15.66%, CVE recall 24.17% on a 10-CVE/38-server set. Dual-reviewer labels (κ=0.87) and a 100-deployment human check (98% agreement) are done properly. The public query interface is a practical add-on.\n\nSoft spots, in proportion. The stress-test note is partly right: the 96.89% figure is a union, high-volume low-precision scanners drive a lot of it, and they report unweighted average precision over stratified alerts rather than volume-weighted or server-level precision on the union. That is a real sampling mismatch, not fatal. Even if you reweight, the low Jaccard and the fact that several high-precision scanners have near-zero recall already show the outputs are not interchangeable ground truth. The CVE set is small and skewed; they say so. Deployment bias toward hermetic servers is real and stated. Free parameters (build attempts, timeouts, star threshold, fixed Qwen backend) are disclosed. Citation pattern is fine; they engage prior static and dynamic MCP work without padding.\n\nThis is for people building or measuring agent tooling and MCP security, not for a general CS audience. Math is light (descriptive stats, Jaccard, precision/recall); data and operational definitions carry the paper. I would bring it to reading group, cite the corpus and the scanner-reliability numbers, and send it to peer review. It deserves referee time; the main ask is tighter linkage between the union statistic and the precision estimate, not a rewrite.","headline":"Large runtime MCP corpus plus a multi-scanner reliability audit that actually lands; the 96.89% headline is a union statistic and the precision sample is imperfectly matched to it, but the unreliability conclusion still holds.","tokens_in":29165,"tokens_out":637,"would_cite":true,"duration_ms":6282,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"MCP security scanners flag nearly all runtime servers as risky, but those alerts are not reliable enough for ecosystem claims.","keywords":["Model Context Protocol","MCP servers","agent security","security scanners","runtime analysis","scanner reliability","MCPZoo"],"falsifier":"A larger independent re-labeling of scanner alerts, or a substantially larger CVE-matched corpus, that showed average precision well above 80 percent, recall well above 70 percent, and high cross-scanner agreement would overturn the claim that scanners are unreliable for ecosystem-level conclusions.","tokens_in":29081,"feed_emoji":"🔒","tokens_out":939,"duration_ms":17417,"temperature":0.7,"pith_summary":"The paper argues that real-world claims about Model Context Protocol (MCP) server security have been built on shaky measurements. Because large numbers of live MCP servers did not previously exist for testing, researchers and practitioners leaned on scanners run on small samples, without knowing how often those scanners were right. The authors build MCPZoo by turning tens of thousands of public repositories into running services through a multi-agent loop that infers environments, deploys containers, repairs failures from logs, and accepts only servers that complete real protocol handshakes and tool listings. On more than 37,000 interactable servers, eight scanners together flag about 97 percent as risky; yet manual review finds average alert precision under 50 percent, scanners rarely agree with each other, and they catch only about a quarter of known CVE cases. The intended takeaway is not that MCP servers are safe, but that current scanning practice cannot yet support ecosystem-level security conclusions; a public query interface is released so users can inspect multi-scanner reports without treating any single alert as ground truth.","feed_headline":"Scanners flag 97% of MCP servers; under half of alerts hold","feed_subtitle":"A 37k-server runtime study finds ~46% precision and almost no agreement across tools.","key_machinery":"MCPZoo: a multi-agent deployment framework (Generation, Verification, and Diagnosis agents) that converts in-the-wild repositories into Dockerized services, validates them with real MCP protocol interactions (stdio, SSE, Streamable HTTP), and yields 64,611 unique servers of which 37,288 support dynamic analysis.","core_discovery":"On 37,288 interactable MCP servers, existing scanners report 96.89 percent as risky, yet those signals are unreliable: a stratified manual sample yields only 45.53 percent average precision, average pairwise Jaccard agreement across scanners is 15.66 percent, and scanners recover only 24.17 percent of a CVE-based ground-truth set. The paper therefore reframes the headline from “MCP servers are unsafe” to “current MCP security scanners are not yet reliable enough for ecosystem-level security claims.”","pith_inferences":["Agent platforms that auto-install popular MCP servers may systematically over-block or under-protect users depending on which scanner they trust.","Markets that re-list near-identical template replicas will amplify both shared real flaws and clustered scanner false positives.","The same multi-agent deploy-and-verify loop could serve as a continuous-integration gate for MCP package registries.","Closing the precision–recall gap may require hybrid static data-flow analysis plus constrained dynamic probing rather than either style alone."],"forward_implications":["Near-97 percent scanner-flag rates should not be read as evidence that nearly all MCP servers are vulnerable.","Ecosystem security measurement needs runtime-validated servers and validated scanners, not metadata-only checks on small curated sets.","Template-driven duplication and weak out-of-the-box deployment practices multiply the instances that inherit the same flaws.","Practical risk triage should surface multi-scanner agreement and validation status rather than treat single-scanner alerts as confirmed vulnerabilities.","Future scanners need evidence grounded in reachable runtime behavior, not capability keywords or unvalidated model inference alone."],"fun_headline_variants":["Scanners flag 97% of MCP servers; under 50% of alerts true","37k runtime MCP servers show scanner precision at 46%","MCP scanners agree on just 16% of risks across tools","Scanners recover only 24% of CVE ground truth on MCP","MCP security claims rest on unreliable scanner signals"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The unreliability claim depends on a manual sample of roughly 100 flagged servers plus a ground-truth set of only 10 CVEs on 38 servers being representative of scanner precision and recall at full ecosystem scale.","fun_headline_variants_meta":{"raw":{"variants":["Scanners flag 97% of MCP servers; under 50% of alerts true","37k runtime MCP servers show scanner precision at 46%","MCP scanners agree on just 16% of risks across tools","Scanners recover only 24% of CVE ground truth on MCP","MCP security claims rest on unreliable scanner signals"]},"model":"grok-4.5","effort":"low","cost_usd":0.00603,"raw_usage":{"total_tokens":1665,"prompt_tokens":894,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":60300000,"prompt_tokens_details":{"text_tokens":894,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":700,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":894,"tokens_out":71,"duration_ms":8411,"temperature":1.0,"reasoning_tokens":700,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T07:09:02.100452+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A larger independent re-labeling of scanner alerts, or a substantially larger CVE-matched corpus, that showed average precision well above 80 percent, recall well above 70 percent, and high cross-scanner agreement would overturn the claim that scanners are unreliable for ecosystem-level conclusions.","supporting_citations":[],"review_version":1}