Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Most MCP-enabled apps let the language model run enabled tools without a human approval step.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:47 UTC pith:7TVW34H2

load-bearing objection First large-scale MCPApp dataset and taxonomy; solid descriptive study, but the headline security number (37.2% approval-gated) leans on static evidence retrieval that needs a false-negative audit. the 3 major comments →

arxiv 2607.25635 v2 pith:7TVW34H2 submitted 2026-07-28 cs.SE

An Empirical Study of Model Context Protocol Applications

Muhammad Hamza Arshad Majeed , May Mahmoud , Sarah Nadi This is my paper
classification cs.SE
keywords Model Context ProtocolMCPAppsdependency managementhuman-in-the-loop controlsconfiguration taxonomyLLM-assisted classificationtool invocation securityopen-source software
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks how AI applications that consume MCP servers integrate those servers as dependencies, and answers with a large-scale study of 1,723 open-source MCPApps. It finds partial convergence: 85.2% declare servers in configuration files and 81.1% communicate through an official SDK, but no naming convention for those files has emerged. The sharpest result is in human oversight: 90.8% log tool activity and 77.2% have enable/disable controls, yet only 37.2% put a blocking approval step before tool execution. In most MCPApps, once a tool is enabled, the LLM's request alone triggers execution. The paper's contribution is a taxonomy (MCPAppTax) and an LLM-assisted classification pipeline that lets the whole population be characterized consistently.

Core claim

Using a manually derived taxonomy and an LLM-assisted classifier over 1,723 repositories found on open-source code hosting, the paper establishes that MCP servers behave like dependencies that lack the standardization package managers provide. It reports that 85.2% of MCPApps configure servers in files (most commonly mcp.json, used in 30.7% of file-configured apps), 81.1% use an official SDK for client-server communication, and the remaining 18.9% implement JSON-RPC communication themselves. On human-in-the-loop controls, logging is near-universal (90.8%) and allowed lists are common (77.2%), but only 37.2% of MCPApps require explicit user approval before a tool executes. The paper interpret

What carries the argument

The central object is MCPAppTax, a taxonomy with five dimensions: configuration type (file, database, or both), configuration file or database name, communication source (official SDK vs self-managed client), use-case domain, and domain specificity; the human-in-the-loop dimension is decomposed into three binary mechanisms: approval required, allowed list, and logging. The machinery that carries the argument is a derive-then-scale pipeline: a small human-labeled sample yields the taxonomy, and an LLM-assisted classifier applies it to the full dataset, with evidence retrieval selecting MCP-relevant files and returning 'Unsure' when evidence is insufficient. It maps every app to a consistent v

Load-bearing premise

The percentages stand or fall on the classifier's accuracy: labels like 'approval required' are assigned only when static source evidence shows a blocking step, so any approval gate hidden in runtime UI, in a dependency, or in a framework unfamiliar to the keyword retrieval could be missed, and the 37.2% figure could shift.

What would settle it

Take a random sample of, say, 100 repositories labeled 'no approval required' and run them in a controlled environment, then trigger a tool call and record whether any user-approval dialog or blocking permission check appears before execution; if more than a few percent do gate execution, the 37.2% figure is an underestimate. Conversely, inspecting apps labeled 'approval required' for a reachable bypass would test the opposite direction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If 85.2% store server declarations in files but with no naming convention, tooling that audits MCP dependencies cannot rely on a fixed manifest location, so automated dependency scanning will need content-based discovery.
  • If 81.1% use official SDKs, SDK conformance is the mainstream path, but the 18.9% self-managed clients constitute a long tail of custom JSON-RPC implementations that need spec-conformance scrutiny.
  • If only 37.2% gate execution behind approval, then for most apps the only control point is pre-session enable/disable, not runtime intervention, so prompt-injection or compromised-server scenarios can execute tools without user awareness.
  • If 20.0% of apps have neither an allowed list nor an approval gate, a non-trivial minority has no structural control at all beyond logging or none.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the absence of a config-file naming convention persists, ecosystem-level tooling such as vulnerability scanners, dependency graphs, and CI audit steps will need to infer manifests from content; the paper's data imply that any tool assuming mcp.json will miss roughly two-thirds of file-configured apps.
  • Inference: The 37.2% approval-gate figure likely overstates runtime protection, since static analysis can miss approval dialogs implemented in UI frameworks, but it could also understate it if keyword-based evidence retrieval misses gates in dependencies; a dynamic test harness that exercises tool calls would resolve which.
  • Inference: The taxonomy invites a prospective standard: a minimal MCP dependency manifest plus a mandatory approval hook, analogous to package-manager lockfiles; the paper's numbers suggest a natural target for such a standard.
  • Inference: The study's snapshot of the ecosystem may be early; as MCP matures, the self-managed client share and the approval-gate share could shift as SDK quality and high-profile security incidents reshape developer practice.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a large-scale empirical study of 1,723 MCP-enabled AI applications (MCPApps) mined from GitHub. The authors derive a taxonomy (MCPAppTax) from a manual analysis of 50 repositories and then apply it using an LLM-assisted classification pipeline to characterize three integration dimensions: configuration of MCP servers, client communication/SDK use, and human-in-the-loop (HITL) controls. The headline findings are that 85.2% of MCPApps configure servers via files with no dominant naming convention, 81.1% use an official MCP SDK, and 37.2% implement a blocking approval gate before tool execution, implying that in 62.8% of MCPApps an enabled tool can be invoked without explicit user approval. The paper also reports domain distributions and combinations of HITL mechanisms, and discusses dependency-management and security implications.

Significance. If the reported percentages are accurate, this is a valuable first host-centric, large-scale characterization of the MCP ecosystem. The paper's strengths include a publicly released dataset and pipeline, a manually derived taxonomy with a reported reliability measure, conservative handling of insufficient evidence via an 'Unsure' label, and explicit discussion of validity threats, including the general risks of LLM-assisted software engineering. The security-relevant claim about the prevalence of ungated tool execution is timely and actionable for the MCP community. However, the central approval-gate measurement rests on static evidence retrieval, and the current validation is too small to bound the false-negative rate on the 'No' class, which is the load-bearing quantity for the paper's security framing.

major comments (3)
  1. [§IV-B-d, §IV-C, Table IV] The 'Approval Required' label is assigned only when the retrieved files 'clearly show a blocking user-approval step' (§IV-B-d). The HITL evidence-retrieval step relies on keyword hits and file-selection priority based on content containing 'mcp' (§IV-B-a/b). Approval logic implemented in a dependency, a dynamically generated UI dialog, or an MCP host library will not appear in the selected evidence files, so such repositories will be labeled 'No'. The validation sets are 50 repositories for the taxonomy and 30 for spot-checking; neither is large enough to estimate the false-negative rate on the 'No' class, which constitutes 62.8% of the resolved approvals. The paper's own §VII-a concedes that static analysis cannot observe runtime behavior. Because the 'only 37.2% gate tool execution' result is the paper's central security claim, I request a targeted audit of repositories labeled 'Approv
  2. [Abstract and §VI-A-c] The paper states categorically that in 62.8% of MCPApps 'the LLM is able to invoke any enabled tool unconditionally' and that 'no approval is required before a tool executes.' These conclusions are stronger than what the measurement supports: the pipeline detects absence of evidence in the retrieved source files, not proof of absence of an approval mechanism. Given the static-analysis limitation acknowledged in §VII-a, the conclusion should be phrased as 'no source-level evidence of a blocking approval gate was found' unless the targeted audit requested above is performed. This is a claim-strength issue directly tied to the paper's main contribution.
  3. [§IV-C] The reported accuracies (98.3% on the 50-repo ground truth and 96.5% on the 30-repo spot-check) are computed over resolved labels after excluding 'Unsure' responses. No confidence intervals or class-conditioned precision/recall are reported. For a binary label at 37.2% 'Yes', even a modest error concentrated on the 'No' side can shift the headline percentage by several points. I recommend reporting per-class recall/precision for the three HITL labels and confidence intervals for the main percentages, or at least making the uncertainty visible in Table IV.
minor comments (5)
  1. [Throughout] 'Y AML' should be 'YAML' (e.g., §IV-A-1, Listing 1 description, §V-B).
  2. [Introduction] The word 'Aritificial Intelligence' should be 'Artificial Intelligence'.
  3. [§V-D] The phrase 'the the three mechanisms' contains a duplicated article; should read 'the three mechanisms'.
  4. [§IV-C] The sentence 'to 100.0% (Communication Source, Logging the full dataset.' appears truncated and should be completed: likely '... Logging)' followed by a new sentence about applying the pipeline.
  5. [§III-C] The transition from 1,727 LLM-classified MCPApps to 1,726 after the 30-repo validation and then to 1,723 after excluding deleted/private repos is understandable but could be stated more explicitly to avoid confusion about which count is used in the final results.

Circularity Check

0 steps flagged

No significant circularity: the study is an empirical measurement pipeline, not a derivation whose outputs are fixed by its inputs.

full rationale

The paper makes no formal derivation; it presents an empirical measurement study. The central percentages (85.2% file-based configuration, 81.1% official SDK use, 37.2% blocking approval gates) are counts computed from LLM-assisted labels applied to 1,723 repositories, where the taxonomy (MCPAppTax) was first derived manually from 50 sampled repositories and then applied at scale. This is a standard derive-then-scale inductive coding design, not a case where a fitted parameter is later renamed as a prediction: the labels are produced from code evidence and independently validated against human ground truth (68-repo identification set, 50-repo taxonomy set, 30-repo spot check, per-dimension accuracy 93.1–100%). The only load-bearing assumption is the accuracy of the LLM classification pipeline, and the paper explicitly acknowledges this as a threats-to-validity concern (§VII-b, citing LLM-assisted-SE threats [33], [34], and §VII-a, noting static analysis cannot observe runtime behavior). That is a correctness/validity risk, not circularity. There is no self-citation chain, no uniqueness theorem imported from the authors' own prior work, no ansatz smuggled in via self-citation, and no equation in which an input is defined in terms of an output. The paper even reports a limitation that some rare configuration patterns fall outside the taxonomy (Section VI-B), which is consistent with honest empirical reporting rather than a circular argument. Therefore, no circular step meets the evidentiary bar required by the analysis guidelines.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper's contribution is empirical measurement; its central percentages rest on methodological thresholds and the untested-at-scale accuracy of the LLM classifier. No free parameters are fitted in a scientific equation, but the thresholds (stars, size, file counts) shape the population, and the taxonomy categories are author-defined.

free parameters (3)
  • minimum GitHub stars threshold = 10 stars
    Hand-chosen filter to remove toy projects; defines the study population and could affect all reported percentages.
  • minimum repository size threshold = 10 KB
    Hand-chosen filter to remove trivial repositories; shapes the population in the same way as the star threshold.
  • evidence file count per LLM query = 20 files
    Chosen during model-selection experiments; reducing to 10 files reduced recall, so 20 was adopted (§IV-B-e).
axioms (4)
  • domain assumption The MCP host/client distinction from the official MCP documentation is the correct unit of analysis (hosts = MCPApps).
    Used throughout, Section II. If 'client' in practice includes SDKs and servers, the dataset composition would differ.
  • domain assumption GitHub search queries, star/size filters, and the January 2024–June 2026 update window capture a representative sample of MCPApps.
    Section III-A. The paper acknowledges GitHub-only and snapshot limitations in Section VII-c.
  • domain assumption Static source code analysis can reliably detect runtime HITL behavior (approval gates, allowed lists, logging).
    Section VII-a: 'we measure integration practices through static source code analysis rather than runtime behavior'.
  • ad hoc to paper The LLM classification pipeline labels the full dataset with accuracy close to the small-sample validation.
    Section IV-B/C and VII-b. This is the load-bearing unproven premise behind the headline percentages.

pith-pipeline@v1.3.0-alltime-deepseek · 15787 in / 9640 out tokens · 95451 ms · 2026-08-01T01:47:53.997094+00:00 · methodology

0 comments
read the original abstract

The Model Context Protocol (MCP) standardizes how large language model applications communicate with external tools, but leaves the application side unspecified: unlike traditional dependencies resolved through package managers, developers integrating MCP servers face no conventions for configuration, communication, or human oversight. This ecosystem is also under-researched, with existing work focused on servers rather than the applications consuming them. We conduct a large-scale study of 1,723 MCPApps mined from GitHub. We first derive MCPAppTax from a representative sample, then use an LLM-assisted pipeline to apply it across the full dataset, characterizing server integration across configuration, SDK use, and human-in-the-loop mechanisms. Our results show that the ecosystem has converged on some practices but not others: most MCPApps configure servers using files (85.2%) and use an official SDK (81.1%) to communicate with servers, yet no naming convention has emerged for configuration files. Human oversight diverges most, logging (90.8%) and enable/disable controls (77.2%) are common, but only 37.2% gate tool execution behind a blocking approval step, leaving the LLM able to invoke any enabled tool unconditionally in most MCPApps.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 3 linked inside Pith

  1. [1]

    (2024, Nov.) Introducing the Model Context Protocol

    Anthropic. (2024, Nov.) Introducing the Model Context Protocol. [Online]. Available: https://anthropic.com

    2024

  2. [2]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459–9474

    2020

  3. [3]

    Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers,

    M. M. Hasan, H. Li, E. Fallahzadeh, G. K. Rajbahadur, B. Adams, and A. E. Hassan, “Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers,”ACM Transactions on Software Engineering and Methodology, 2025

    2025

  4. [4]

    Model context protocol (MCP): Landscape, security threats, and future research directions,

    X. Hou, Y . Zhao, S. Wang, and H. Wang, “Model context protocol (MCP): Landscape, security threats, and future research directions,” arXiv preprint arXiv:2503.23278, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Selecting third-party libraries: The practitioners’ perspective,

    E. Larios Vargas, M. Aniche, C. Treude, M. Bruntink, and G. Gousios, “Selecting third-party libraries: The practitioners’ perspective,” inPro- ceedings of the 28th ACM Joint European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2020, pp. 245–256

    2020

  6. [6]

    Measuring dependency freshness in software systems,

    J. Cox, E. Bouwers, M. van Eekelen, and J. Visser, “Measuring dependency freshness in software systems,” inProceedings of the 37th International Conference on Software Engineering (ICSE), vol. 2. IEEE, 2015, pp. 109–118

    2015

  7. [7]

    Do developers update their library dependencies? an empirical study on the impact of security advisories on library migration,

    R. G. Kula, D. M. German, A. Ouni, T. Ishio, and K. Inoue, “Do developers update their library dependencies? an empirical study on the impact of security advisories on library migration,”Empirical Software Engineering, vol. 23, no. 1, pp. 384–417, 2018

    2018

  8. [8]

    Architecture overview,

    Anthropic, “Architecture overview,” https://modelcontextprotocol.io/ docs/learn/architecture, 2025, accessed: 2026-06-18

    2025

  9. [9]

    Model Context Protocol Threat Modeling and Analysis of Vulnerabilities to Prompt Injection with Tool Poisoning,

    C. Huang, X. Huang, N. Tran, and A. M. Fard, “Model Context Protocol Threat Modeling and Analysis of Vulnerabilities to Prompt Injection with Tool Poisoning,”Journal of Cybersecurity and Privacy, mar 2026

    2026

  10. [10]

    Typescript sdk for the model context proto- col,

    Model Context Protocol, “Typescript sdk for the model context proto- col,” https://github.com/modelcontextprotocol/typescript-sdk, 2024, ac- cessed: 2026-06-30

    2024

  11. [11]

    Python sdk for the model context protocol,

    ——, “Python sdk for the model context protocol,” https://github.com/ modelcontextprotocol/python-sdk, 2024, accessed: 2026-06-30

    2024

  12. [12]

    Java sdk for the model context protocol,

    ——, “Java sdk for the model context protocol,” https://github.com/ modelcontextprotocol/java-sdk, 2024, accessed: 2026-06-30

    2024

  13. [13]

    Kotlin sdk for the model context protocol,

    ——, “Kotlin sdk for the model context protocol,” https://github.com/ modelcontextprotocol/kotlin-sdk, 2024, accessed: 2026-06-30

    2024

  14. [14]

    C# sdk for the model context protocol,

    ——, “C# sdk for the model context protocol,” https://github.com/ modelcontextprotocol/csharp-sdk, 2024, accessed: 2026-06-30

    2024

  15. [15]

    Swift sdk for the model context protocol,

    ——, “Swift sdk for the model context protocol,” https://github.com/ modelcontextprotocol/swift-sdk, 2024, accessed: 2026-06-30

    2024

  16. [16]

    Rust sdk for the model context protocol,

    ——, “Rust sdk for the model context protocol,” https://github.com/ modelcontextprotocol/rust-sdk, 2024, accessed: 2026-06-30

    2024

  17. [17]

    Nextchat,

    ChatGPTNextWeb, “Nextchat,” https://github.com/ChatGPTNextWeb/ NextChat, 2025, accessed: 2026-06-30

    2025

  18. [18]

    Chatmcp,

    daodao97, “Chatmcp,” https://github.com/daodao97/ChatMCP, 2025, ac- cessed: 2026-06-30

    2025

  19. [19]

    A large-scale dataset of mcp implementations on github,

    B. Toeppe, A. Barrak, and E. Ksontini, “A large-scale dataset of mcp implementations on github,” inProceedings of the 23rd International Conference on Mining Software Repositories (MSR) - Data and Tool Track, ser. MSR ’26. IEEE / ACM, 2026

    2026

  20. [20]

    gemini-cli,

    google-gemini, “gemini-cli,” https://github.com/google-gemini/ gemini-cli, 2025, accessed: 2026-06-30

    2025

  21. [21]

    IBM, “mcp,” https://github.com/IBM/mcp, 2025, accessed: 2026-06-30

    2025

  22. [22]

    Awesome mcp clients,

    punkpeye, “Awesome mcp clients,” https://github.com/punkpeye/ awesome-mcp-clients, 2025, accessed: 2026-06-30

    2025

  23. [23]

    nanbingxyz, “5ire,” https://github.com/nanbingxyz/5ire, 2025, accessed: 2026-06-30

    2025

  24. [24]

    Block, “goose,” https://github.com/aaif-goose/goose, 2025, accessed: 2026-06-30

    2025

  25. [25]

    learnopencv,

    spmallick, “learnopencv,” https://github.com/spmallick/learnopencv, 2025, accessed: 2026-06-30

    2025

  26. [26]

    johnrobinsn, “askit,” https://github.com/johnrobinsn/askit, 2025, ac- cessed: 2026-06-30

    2025

  27. [27]

    mario-andreschak, “Flujo,” https://github.com/mario-andreschak/ FLUJO, 2025, accessed: 2026-06-30

    2025

  28. [28]

    NitroRCr, “Aiaw,” https://github.com/NitroRCr/AIaW, 2025, accessed: 2026-06-30

    2025

  29. [29]

    code-cli,

    autohandai, “code-cli,” https://github.com/autohandai/code-cli, 2025, ac- cessed: 2026-06-30

    2025

  30. [30]

    Astrbot,

    AstrBotDevs, “Astrbot,” https://github.com/AstrBotDevs/AstrBot, 2025, accessed: 2026-06-30

    2025

  31. [31]

    ollama-mcp-example,

    kirillsaidov, “ollama-mcp-example,” https://github.com/kirillsaidov/ ollama-mcp-example, 2025, accessed: 2026-06-30

    2025

  32. [32]

    http-mcp-bridge,

    NCC Group, “http-mcp-bridge,” https://github.com/nccgroup/ http-mcp-bridge, 2025, accessed: 2026-06-30

    2025

  33. [33]

    Evaluation guidelines for empirical studies in software engineering involving large language models,

    S. Balteset al., “Evaluation guidelines for empirical studies in software engineering involving large language models,”arXiv preprint, 2025

    2025

  34. [34]

    Genai is no silver bullet for qualitative research in software engineering,

    N. A. Ernst and C. Treude, “Genai is no silver bullet for qualitative research in software engineering,”arXiv preprint arXiv:2603.08951, 2026

    Pith/arXiv arXiv 2026

  35. [35]

    A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis,

    Z. Lin, B. Ruan, J. Liu, and W. Zhao, “A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis,”ArXiv, vol. abs/2506.23474, jun 2025

    Pith/arXiv arXiv 2025

  36. [36]

    A Measurement Study of Model Context Protocol,

    H. Guo, Y . Hao, Y . Zhang, M. Xu, P. Lyu, J. Chen, and X. Cheng, “A Measurement Study of Model Context Protocol,”ArXiv, vol. abs/2509.25292, sep 2025

    arXiv 2025

  37. [37]

    How are AI agents used? Evidence from 177,000 MCP tools,

    M. Stein, “How are AI agents used? Evidence from 177,000 MCP tools,” arXiv preprint arXiv:2603.23802, 2026

    arXiv 2026

  38. [38]

    A survey of the Model Context Protocol (MCP): Standardizing Context to Enhance Large Language Models (LLMs),

    A. Singh, A. Ehtesham, S. Kumar, and T. T. Khoei, “A survey of the Model Context Protocol (MCP): Standardizing Context to Enhance Large Language Models (LLMs),”Preprints, apr 2025

    2025

  39. [39]

    Shostack,Threat Modeling: Designing for Security

    A. Shostack,Threat Modeling: Designing for Security. Wiley, 2014

    2014