{"id":"2b3234b1-e671-4a17-b255-da2ecff24923","arxiv_id":"2607.10123","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-stage GitHub mining pipeline yields 2,297 verified MCP client, server, and gateway repositories with role labels, evidence tags, and 83% precision on a manual sample.","lead":"Researchers built a verified dataset of 2,297 real GitHub projects that implement the Model Context Protocol for connecting AI models to tools. The collection, with role labels and code-level evidence, gives a shared baseline for studying how this new AI-tool standard is actually used.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The 83% sample precision and post-hoc exclusion of 90 non-operational repos do not tightly bound residual false positives or role-assignment error on the full 2,297 set.","rationale":"The reader correctly isolates the weakest assumption: that the chosen fingerprints, entry-point patterns, scoring thresholds, and single 170-repo dual-author sample adequately separate operational MCP code from FPs and correctly assign primary roles for the full set. That is precisely the load-bearing concern; the paper’s own numbers (83% precision, 12% FP, post-hoc drop of 90, 5% FN, dual counting of multi-role repos, 36 Unclassified) make the extrapolation non-trivial. No deeper logical break or circularity appears—the pipeline is transparent and the public-release claim is appropriate for an MSR dataset paper—so the verdict remains CONDITIONAL rather than REJECT. A stratified re-audit of the released set would settle whether residual error is material. Confidence stays moderate because the review did not execute the claimed code/data.","tokens_in":7618,"tokens_out":630,"duration_ms":5820,"concrete_test":"Independently re-sample 100 repositories stratified by role and language from the final 2,297 (or the public Zenodo release), re-apply the exact §3.2 scoring rules and dual-author operational check, and compute residual precision plus role-agreement (Cohen’s κ). If residual precision falls below ~80% or role agreement is <0.7, the validated-corpus claim and role tallies need qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that multi-stage evidence checks plus a dual-author review of 170 repos yield a validated operational MCP corpus of 2,297 projects with reliable role labels (1,962 Servers, 1,462 Clients, 80 Gateways). The load-bearing step is the leap from that sample (83% genuine, 12% FP, 5% FN; Wilson CI [79%,87%]) and the subsequent removal of 90 false matches to the assertion that residual contamination and role misclassification are low enough for the full set to serve as a foundational benchmark. Section 2.4 notes that most errors came from language-specific conventions and incomplete manifests; the rule-based scores in §3.2 (Sc≥3 & Ss≤1 → Client; Ss≥3 & Sc≤1 → Server; Sg≥4 & others <2 → Gateway, plus editor override) and the dual-role counting are author-chosen thresholds never re-validated after refinement. Multi-role repos are counted in both tallies without a clear primary-role policy, and the 36 Unclassified cases plus the 5% FN rate imply under-coverage whose effect on the reported language/role distributions is unquantified. If residual FPs or role errors remain concentrated in the dominant Python/TypeScript hybrid classes, the headline counts and “hybrid architectures most common” claim weaken.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper constructs and releases a repository-centric dataset of Model Context Protocol (MCP) implementations mined from GitHub. Starting from 3,238 candidates discovered via five REST/GraphQL search queries (Jan 2024–Oct 2025), the authors apply activity filters, multi-stage code-level evidence checks (dependency manifests, directory/entry-point patterns, CI/Docker/Claude configs), and dual-author manual review of a 170-repository sample (83% precision, Wilson 95% CI [79%, 87%]). After removing 90 additional non-operational items they obtain 2,297 verified projects, each tagged with operational role (1,962 Servers, 1,462 Clients, 80 Gateways, 36 Unclassified), language, evidence signals, and commit/PR metadata, exported in a reproducible JSONL schema. Python and TypeScript dominate; hybrid client–server repositories are noted as common.","tokens_in":8068,"tokens_out":1034,"duration_ms":18167,"significance":"If the residual false-positive rate and role labels are reliable, this is a useful foundational artifact for the emerging MCP ecosystem: it is the first large-scale, code-evidence-driven (rather than registry-driven) corpus, supplies transparent multi-layer verification tags, dual-author sample validation, and public data/code. Those strengths directly enable follow-on work on security, maintainability, evolution, and interoperability that prior marketplace listings cannot support. The contribution is therefore of clear interest to the MSR community even if the absolute counts require modest qualification.","major_comments":[{"comment":"§2.4 reports 83% precision on a 170-repo dual-author sample drawn from the 2,387 automatically verified set, then removes 90 further non-operational repositories to reach 2,297. No second independent sample is drawn after the rule refinement, so the residual false-positive rate on the final corpus is not re-estimated. Because the central claim is that the released set constitutes a ‘validated’ foundational benchmark, a post-refinement precision (or at least a quantified upper bound on residual contamination) is load-bearing and currently missing.","section":"§2.4"},{"comment":"§3.2 defines role assignment via author-chosen numeric thresholds (Sc≥3 & Ss≤1 → Client; Ss≥3 & Sc≤1 → Server; Sg≥4 & others <2 → Gateway) plus a client-override rule for editor signals. These thresholds and the dual-role counting policy are never validated against the manual sample (which assessed only presence of MCP, not role correctness). Consequently the headline tallies (1,962 Servers / 1,462 Clients) and the abstract claim that ‘hybrid architectures [are] the most common design pattern’ rest on unmeasured classification error; a modest role-accuracy audit on the same or a fresh sample is required.","section":"§3.2"},{"comment":"The abstract and §3 assert that hybrid architectures are the most common pattern, yet the paper never reports the cardinality of the dual-role intersection. Because repositories may be counted in both Server and Client tallies, the raw numbers alone do not establish that hybrids outnumber pure clients or pure servers. An explicit contingency table (pure-client / pure-server / hybrid / gateway / unclassified) is needed to substantiate the claim.","section":"Abstract / §3.2"}],"minor_comments":[{"comment":"Table 1 column headers are slightly ambiguous (‘Total repositories (response)’ vs ‘(Kept)’); a short footnote clarifying that ‘Kept’ already excludes duplicates/archived items would help.","section":"Table 1"},{"comment":"Figure 3 schema diagram is dense; labeling the foreign-key edges or providing a one-sentence legend would improve readability.","section":"Figure 3"},{"comment":"The nine-month inactivity cutoff (§2.2) and the exact list of dependency fingerprints are free parameters; listing them in an appendix or config file would aid exact reproduction.","section":"§2.2–2.3"},{"comment":"A few minor wording issues: ‘market centered efforts’ → ‘market-centered efforts’; ‘point in time metadata’ → ‘point-in-time metadata’.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid dataset contribution for MSR and should be publishable after the three major points are addressed. Novelty relative to the concurrent MCPCorpus (Lin et al.) is adequately differentiated (repo-first vs marketplace-first), so I see no citation or priority concern. Scope fit is excellent."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean, useful dataset paper for a fast-moving niche. What is new is a repository-first MCP corpus built from GitHub code evidence rather than marketplace listings: 3,238 candidates down to 2,297 validated projects, with multi-layer signals (manifests, entry points, directory patterns, CI/Docker/Claude config), role labels (servers/clients/gateways), evidence tags, and a public JSONL schema plus claimed Zenodo release. That is real infrastructure for anyone studying agent tooling, protocol security, or maintainability.\n\nThey do the MSR basics well. Discovery queries and filters are explicit. Verification stages are concrete. Dual-author review of a 170-repo sample at 95% CI reports 83% precision (Wilson [79%, 87%]), then they remove 90 more non-operational items. Role scoring rules and the six gateway subtypes are written down. Language and activity stats (Python/TypeScript dominance, small teams, commit medians) are descriptive but grounded. Citations to Hasan et al. and Lin et al. (MCPCorpus) correctly position the work as complementary, not redundant. No circular math; free parameters (score thresholds, nine-month cutoff, time window) are visible.\n\nSoft spots are the ordinary ones for this genre, not load-bearing collapses. The leap from the sample plus post-hoc cleanup to “foundational benchmark” residual error on the full set is not tightly bounded; thresholds were not re-validated after refinement, multi-role repos are double-counted without a primary-role policy, and the 5% FN rate plus 36 Unclassified cases leave under-coverage unquantified. If residual FPs concentrate in the big Python/TS hybrid classes, the headline distributions soften. Inter-rater detail is thin. None of that kills the resource; it just means users should treat labels as high-quality heuristics and re-check for high-stakes work.\n\nWho it is for: empirical SE people and security researchers who need a starting corpus of real MCP clients/servers/gateways. It deserves a serious referee at MSR. I would engage with the data, cite the construction method, and bring it to reading group as a practical example of transparent mining for an emerging protocol. Accept for review; expect requests for fuller release artifacts and clearer multi-role handling.","headline":"Solid MSR dataset paper: first repo-first, evidence-tagged MCP corpus with roles; useful infrastructure, usual heuristic limits.","tokens_in":8635,"tokens_out":568,"would_cite":true,"duration_ms":5690,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A code-evidence pipeline yields 2,297 verified MCP projects on GitHub, mostly Python and TypeScript servers and clients.","keywords":["Model Context Protocol","MCP","dataset","GitHub","repository mining","open source software","GraphQL","gateways"],"falsifier":"An independent re-label of a fresh stratified sample of several hundred repositories (or of the published 170) that finds substantially lower precision than 83 percent, or that systematically reassigns roles under the same evidence rules, would overturn the claimed reliability of the final 2,297-project set.","tokens_in":8487,"feed_emoji":"📦","tokens_out":894,"duration_ms":9118,"temperature":0.7,"pith_summary":"The Model Context Protocol is becoming a shared way for language-model agents to discover and call external tools, but until now most maps of the ecosystem came from marketplace listings rather than from the code itself. This paper builds the first large repository-first dataset by searching GitHub for MCP-related projects, then confirming each one with multi-stage evidence: dependency fingerprints, directory and entry-point patterns, and deployment configuration. After filtering inactive and documentation-only repos and removing a further set of tutorial templates found by manual review, the authors release 2,297 validated operational projects classified as clients, servers, or gateways. A dual-author check on a representative sample gives 83 percent precision. The resulting public JSONL resource, complete with evidence tags and commit history, is intended as a reproducible benchmark so researchers can study how MCP is actually implemented, connected, and maintained rather than how it is advertised.","feed_headline":"2,297 verified MCP projects mined from GitHub code","feed_subtitle":"Evidence checks and role labels turn raw search hits into a reusable benchmark for agent-tool ecosystems","key_machinery":"Multi-stage evidence verification plus rule-based role scoring: repositories must show MCP-related dependencies, directory/entry-point patterns, or integration configs; each then receives numeric client/server/gateway scores from strong (source/manifest) and weak (metadata) signals, with fixed thresholds and a client override for editor plugins determining the primary role.","core_discovery":"A hybrid GitHub REST/GraphQL discovery pipeline followed by multi-stage code-level evidence checks and a targeted exclusion of non-operational templates produces a final dataset of 2,297 validated MCP implementations (from 3,238 candidates), role-classified as 1,962 servers, 1,462 clients, 80 gateways and 36 unclassified, with 83 percent precision on a dual-author manual sample and with Python and TypeScript as the dominant languages.","pith_inferences":["Because the pipeline is fully documented and the evidence is stored, community re-runs can detect drift as MCP libraries and naming conventions evolve.","The high share of dual client-server repositories suggests that many early MCP projects are self-contained demos rather than pure protocol endpoints, which may shape how interoperability tests should be designed.","Gateway rarity (only 80) implies that mediation layers are still scarce; security and routing research may therefore need synthetic or constructed examples until more real gateways appear."],"forward_implications":["Researchers can ground claims about MCP security, maintainability, and evolution in repository-verified code rather than marketplace listings alone.","Language and role statistics (Python/TypeScript dominance, hybrid client-server codebases, gateway subtypes) become measurable baselines for ecosystem studies.","The public JSONL schema with evidence tags and commit/PR history supports longitudinal tracking of how MCP implementations change.","Future work on integration, connectivity, and compatibility can start from a shared, reproducible ground-truth set of operational projects."],"fun_headline_variants":["2,297 verified MCP projects from GitHub evidence checks","First large-scale MCP dataset: 2,297 role-labeled projects","Hybrid pipeline yields 2,297 validated MCP implementations","GitHub MCP benchmark: 2,297 projects after template exclusion","2,297 MCP clients, servers and gateways mined and classified"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The chosen dependency fingerprints, directory patterns, and scoring thresholds, together with one 170-repository dual-author sample, are enough to separate real operational MCP code from false positives and to assign primary roles correctly for the whole set.","fun_headline_variants_meta":{"raw":{"variants":["2,297 verified MCP projects from GitHub evidence checks","First large-scale MCP dataset: 2,297 role-labeled projects","Hybrid pipeline yields 2,297 validated MCP implementations","GitHub MCP benchmark: 2,297 projects after template exclusion","2,297 MCP clients, servers and gateways mined and classified"]},"model":"grok-4.5","effort":"low","cost_usd":0.005754,"raw_usage":{"total_tokens":1541,"prompt_tokens":825,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":57540000,"prompt_tokens_details":{"text_tokens":825,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":626,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":825,"tokens_out":90,"duration_ms":5208,"temperature":1.0,"reasoning_tokens":626,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T14:07:24.805801+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"An independent re-label of a fresh stratified sample of several hundred repositories (or of the published 170) that finds substantially lower precision than 83 percent, or that systematically reassigns roles under the same evidence rules, would overturn the claimed reliability of the final 2,297-project set.","supporting_citations":[],"review_version":1}