Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that real-world MCP server faults cluster into five empirically derived categories, and a practitioner survey confirms all five occur in practice.

desk verdict Useful first taxonomy of MCP server faults, but the 'comprehensive' framing outruns the evidence; still worth a serious referee. read the letter →

arxiv 2603.05637 v2 pith:DGD3Z35D submitted 2026-03-05 cs.SE

Mina Taraghi , Mohammad Mehdi Morovati , Foutse Khomh This is my paper · ORCID
classification cs.SE
keywords ModelContextProtocolMCPserverssoftwarefaulttaxonomyLLM-basedsystemsempiricalengineeringissueminingtoolintegrationconfigurationfaults
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This study tries to establish that the faults that actually break Model Context Protocol (MCP) servers are not random or idiosyncratic: they cluster into five recognizable categories. Analyzing 407 bug-related closed issues from 385 open-source MCP server projects, the authors derive a taxonomy whose largest categories are server/tool configuration, server/host configuration, and server setting, followed by documentation and general programming. A survey of 41 MCP practitioners reports encountering every category in practice, which the authors take as evidence the taxonomy is complete. If right, the taxonomy gives developers a checklist of where MCP systems fail and gives researchers a baseline for MCP-specific testing and repair tools.

What carries the argument

The load-bearing object is the taxonomy itself, anchored in MCP's host-client-server architecture. Faults are classified according to where in the MCP stack the break happens: in the server's own setting, in how the server wires tools, in how the host connects to the server, in documentation, or in ordinary programming errors. The taxonomy works by taking issue reports and assigning each to a category based on the component boundary where the fault manifests, which is what lets the study compare fault types and separate MCP-specific problems from generic ones.

What would settle it

Label a fresh random sample of 200 closed bug issues from MCP server projects written in non-Python SDKs using the paper's codebook. If a substantial fraction (say more than 15-20%) of MCP-related bugs fall outside the five categories, or if the category distribution shifts dramatically, the taxonomy's completeness and generalizability claims fail.

Watch

Extended reading notes

Core claim

The central claim is that the first empirically grounded taxonomy of MCP server faults consists of five high-level categories: Server Setting (27.45% of labeled issues), Server/Tool Configuration (31.74%), Server/Host Configuration (28.64%), Documentation (6.92%), and General Programming (5.25%). The study also finds that MCP-specific faults differ measurably from non-MCP faults: they attract more comments and more comments per collaborator, yet are fixed faster and by developers with less overall experience. The authors interpret the survey results as confirming that all taxonomy categories occur in practice, with tool response handling and documentation reported most frequently and tool di

Load-bearing premise

The claim rests on the assumption that the 407 issues selected by the LLM-assisted classification and clustering pipeline faithfully represent real MCP faults; the pipeline's accuracy was checked on only 40 hand-labeled issues, and clustering-relatedness was judged on 20 issues per cluster, so sampling or labeling errors could reshape the taxonomy.

Editorial extensions

If this is right

  • MCP server testing efforts should target configuration and integration seams—dependencies, tool registration, host connection settings—rather than only tool logic.
  • Issue triage tools for MCP repositories can use the five categories as a checklist to route and prioritize bug reports.
  • MCP-specific fault patterns being distinct from generic bugs means automated debugging and repair should be trained on MCP-aware data, not generic software faults.
  • The observed short fix times and lower fixer experience for MCP faults suggest a small pool of domain specialists handles them; improving documentation and defaults could reduce that bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The taxonomy's three dominant categories are all configuration seams, suggesting that MCP's fragility lives at integration boundaries rather than in the core protocol logic; a testable consequence is that configuration-focused static analysis would catch most avoidable failures.
  • The survey's finding that tool discovery/registration is rare but rated critical implies that availability failures outweigh frequency when prioritizing MCP reliability; one could monitor issue trackers to test whether discovery faults cause outsized user impact.
  • Because the sample is Python-only and collected soon after MCP's release, the taxonomy may underrepresent faults that emerge with maturity, such as security issues and version skew across SDKs; extending the same coding to TypeScript SDKs and later time windows is a natural check.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study of faults in Model Context Protocol (MCP) servers. The authors mine 385 GitHub repositories that use the Python MCP SDK, extract 26,821 closed issues, classify them with an LLM (GPT-4o-mini) into bug and non-bug categories, cluster the 3,282 bug issues with BERTopic, and manually identify 407 MCP-related issues. These issues are open-coded to produce a five-category taxonomy: Server Setting, Server/Tool Configuration, Server/Host Configuration, Documentation, and General Programming, with a detailed subcategory structure. The taxonomy is said to be validated by a survey of 41 MCP practitioners, and the paper further compares MCP-related versus non-MCP faults on metrics such as fix time, number of comments, collaborators, and fixer experience. The central claim is that this is the first large-scale, comprehensive taxonomy of real faults in MCP servers, and that all identified categories occur in practice.

Significance. If the taxonomy is sound, this is a valuable contribution to the emerging area of MCP reliability engineering. The paper has clear strengths: the methodology is transparently described, the taxonomy is data-derived rather than imposed a priori, concrete issue examples ground each category, a replication package is promised, and the survey explicitly involves practitioners who did not construct the taxonomy. The statistical analyses report effect sizes, not just p-values, which is good practice. However, the degree to which the taxonomy is 'comprehensive' depends on the recall of the automated filtering pipeline, which is not measured. The small survey and its recruitment from the same repository population limit the strength of the validation. These concerns do not invalidate the taxonomy as a useful initial structure, but they do affect the strength of the central claim and the generalizability of the quantitative proportions.

major comments (4)
  1. [§3.2.2 and §6.2] The MCP-relatedness filter has no measured recall. Clusters are labeled MCP-related by manually reviewing 20 representative issues per cluster, and issues in non-MCP clusters are recovered only if their LLM-generated summary contains the literal token 'MCP'. Because issue titles and summaries often omit the acronym (e.g., 'server crashes on startup' in an MCP-server repository), MCP-specific issues assigned to non-MCP clusters are silently dropped. This directly affects the 407-issue sample, the taxonomy structure, and the category proportions in Fig. 3. The paper's threat-to-validity discussion acknowledges this but does not quantify the risk. Please add a recall estimate (e.g., manually inspect a random sample from non-MCP clusters or from issues whose summaries do not contain 'MCP') or explicitly soften the 'comprehensive' claim to one about the population of issues that are explicitl
  2. [§4.1 and §4.3 (Tables 5 and 6)] The statistical comparison of fault characteristics treats labels as independent units. The paper reports 419 labels from 383 issues after excluding 24 issues, stating that some issues received multiple labels. For metrics such as 'required time to fix,' an issue with two labels contributes to both groups in the Kruskal–Wallis and Dunn tests, violating the independence assumption and potentially inflating significance or distorting effect sizes. Please use a primary label per issue for the RQ3 analysis, or otherwise account for the multi-label structure. This is important because the significant differences reported in Table 6 may not be robust.
  3. [§3.2.3] The open-coding step is the sole basis for the taxonomy, but no inter-rater reliability statistic is reported. The paper states this is because no predefined labels were available. While this is understandable for emergent coding, the central claim would be substantially strengthened by reporting the number of disagreements resolved, a post-hoc agreement check on a random subset using the final codebook, or a more detailed justification of why a reliability measure is impossible. Without this, readers cannot distinguish robust categories from the systematic bias of the two raters. The current description of regular meetings and third-adjudicator resolution is helpful, but it does not quantify reliability.
  4. [§3.3 and §4.2 (Table 4)] The survey is presented as validating the completeness of the taxonomy, but the evidence is weak. There are 41 valid responses, recruited from contributors to the same repositories that produced the issue sample; the response rate is not reported. Table 4 shows that every category is encountered by at least some respondents, but this only confirms that each category is non-empty; it does not establish completeness or external generalizability. The absence of newly suggested fault types from 41 respondents is not strong evidence of coverage. Please reframe the survey as a preliminary qualitative check, report recruitment and response-rate information, and avoid the conclusion that it 'supports the completeness' of the taxonomy.
minor comments (5)
  1. [§7 (Conclusion)] The conclusion says '443 MCP server repositories,' while §3.1 reports 385 repositories after manual checking. This inconsistency should be fixed.
  2. [§4.1] The relationship between 407 inspected issues, 24 excluded issues, and 419 labels should be stated more clearly. The current text says 'Each issue was examined and labeled... resulting in 419 labels' and then mentions 24 exclusions; readers may wonder whether the percentages are computed on labels or issues.
  3. [Figure 2] The methodology figure contains the typo 'MPC-based Repos'; it should read 'MCP-based Repos'.
  4. [Table 4] The table reports prevalence percentages without giving the denominator for each row. Since the number of respondents who answered each category group may differ, include N per row or clarify how missing responses are handled.
  5. [§3.2.2] The paper says HDBSCAN was used to estimate the number of clusters and then KMeans was applied with k=101. It is not clear how the HDBSCAN result maps to 101; a brief description or a sensitivity check would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the taxonomy is derived from manual labeling of issue reports and validated by independent survey respondents.

full rationale

The paper's central deliverable is an empirically grounded taxonomy of faults in MCP servers. The derivation chain is: collect repositories using the MCP Python SDK; filter them; classify closed issues with an LLM selected on 40 hand-labeled examples; cluster bug summaries with BERTopic; manually review 20 issues per cluster plus keyword fallback to identify 407 MCP-related issues; open-code those issues into a codebook; group labels into a hierarchy; and validate via a survey of 41 MCP practitioners. None of these steps defines a taxonomy category in terms of the survey outcome, nor does the survey define the categories: survey respondents were not the taxonomy constructors, were asked whether they had encountered each pre-existing fault type, and were invited to suggest new types. The manual labeling and codebook construction in Section 3.2.3 are independent of the survey validation. Self-citations such as [114], [116], and [117] are methodological (exclusion criteria, metric definitions, taxonomy-construction procedure) rather than load-bearing theoretical premises, and no uniqueness theorem or prior result by the same authors is invoked to force the taxonomy's structure. The main limitations—unmeasured recall in the LLM/clustering filter (acknowledged in Section 6.2 as a risk of false negatives) and recruitment of survey participants from the same repository ecosystem—affect external validity and confidence in completeness, but they are not cases where a 'prediction' is equivalent to an input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No mathematical free parameters or invented entities are introduced. The listed free parameters are data-selection and methodological thresholds that shape the corpus, the 407-issue sample, and the survey validation. The axioms are the assumptions needed to treat mined GitHub issues, LLM-assisted labeling, clustering, manual coding, and a small survey as evidence for a comprehensive taxonomy.

free parameters (6)
  • min_stars_and_forks = 10
    Repositories with fewer than 10 stars and 10 forks were excluded (§3.1), shaping the corpus of MCP servers from which all faults were drawn.
  • date_threshold = 2024-11-01
    Only closed issues after Nov 1, 2024 were kept (§3.1), because MCP was officially introduced then; this truncates the fault history.
  • number_of_clusters_k = 101
    KMeans clustering into 101 clusters guided which issues were considered MCP-related; the taxonomy depends on this grouping (§3.2.2).
  • reviewed_issues_per_cluster = 20
    Raters manually reviewed 20 representative issues per cluster to decide MCP-relatedness; this selection threshold determines the 407-issue sample (§3.2.2).
  • llm_ground_truth_size = 40
    GPT-4o-mini was chosen for issue classification based on 40 manually labeled issues; classifier accuracy is assumed to generalize (§3.2.1, Table 1).
  • survey_sample_size = 41 valid responses
    Completeness validation rests on 41 self-selected respondents from 10,756 invited email addresses (§4.2).
assumptions (6)
  • domain assumption Closed GitHub issues, after LLM triage, are a valid proxy for real MCP server faults.
    The entire taxonomy is mined from closed issues labeled as bugs; issue text may misreport root cause or be environment-specific (§3.2.1, §6.1).
  • domain assumption GPT-4o-mini's issue-classification performance on 40 ground-truth issues (F1≈0.77) generalizes to 26,821 issues.
    No second validation set is reported; misclassification propagates into the bug set and the taxonomy (§3.2.1, Table 1).
  • domain assumption BERTopic clustering accurately separates MCP-related from non-MCP issues so the 407-issue sample is representative.
    Cluster membership and 20-issue manual review determine MCP-relatedness; false negatives are acknowledged in §6.2.
  • domain assumption Open coding by two raters without computing inter-rater agreement yields stable categories.
    Section 3.2.3 states agreement cannot be calculated because no predefined labels; this weakens the reliability of the final taxonomy categories.
  • domain assumption 41 survey respondents are representative of MCP practitioners.
    Self-selected respondents, mostly developers, drawn from the same repository pool; used to claim completeness (§4.2).
  • standard math Non-parametric statistical tests (Kruskal–Wallis, Mann–Whitney, Dunn) with reported effect sizes are appropriate for the metric distributions.
    Used per §4.3 after Shapiro–Wilk rejected normality; standard statistical tools.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy." pith.science (2026). https://pith.science/paper/DGD3Z35D

@misc{pith2026260305637,
  author       = {Pith},
  title        = {Pith review of: Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DGD3Z35D}},
  note         = {Machine review of arXiv:2603.05637}
}
read the original abstract

The rapid adoption of foundation models has significantly expanded the capabilities of software systems, enabling them to perform complex language, reasoning, and interaction tasks that were previously difficult to automate. However, this progress has also introduced novel challenges that were largely absent in previous generations of software. In particular, the increasing integration of foundation models with external tools and resources raises new concerns regarding reliability, security, and robustness. The Model Context Protocol (MCP) has recently been proposed to standardize interactions between AI-based software systems, software tools, and external resources. Despite its growing adoption, there remains limited systematic understanding of real-world faults in MCP-based software systems. In this paper, we present the first large-scale taxonomy of faults in MCP servers, comprising five high-level fault categories derived from empirical evidence. To evaluate the completeness and generalizability of this taxonomy, we conduct a survey of MCP practitioners with diverse roles and experience levels. The results confirm that all identified fault categories occur in practice and reveal distinct characteristics that differentiate MCP-specific faults from non-MCP faults. Overall, this study provides actionable insights for researchers and practitioners by identifying the most error-prone and critical components of MCP-based systems. These insights can inform the development of more robust, reliable, and secure AI-enabled software systems that rely on MCP.

Figures

Figures reproduced from arXiv: 2603.05637 by the authors.

Figure 1
Figure 1. A high-level view of Model Context Protocol (MCP) Architecture [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. High-level view of the used methodology files within defined ranges (using the parameter ‘size:<min>..<max>’) to ensure that each query returned fewer than 1000 results. Since the search API do not return any results for file sizes greater than 350 KB in our case, we divide the entire range from 1 B to 350 KB into intervals of 50 B. This process result in a total of approximately 7,000 API calls. We then extract the… view at source ↗
Figure 3
Figure 3. Taxonomy of MCP server issues (counts in parentheses). [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of faults’ characteristics for the 5 main categories of bugs in terms of metrics including [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]
Figure 5
Figure 5. Figure 5: Comparison of faults’ characteristics for MCP vs. Non-MCP [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

    cs.SE 2026-08 reject novelty 6.0 of 10

    An empirical GitHub mining study finds vLLM is the most adopted LLM serving framework, parallel and memory optimizations dominate, and multi-framework use is rare.

Reference graph

Works this paper leans on

171 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Data Mining: Concepts and Techniques (Third Edition)

    2012. Data Mining: Concepts and Techniques (Third Edition). Morgan Kaufmann, Boston, xxiii–xxix. doi:10.1016/B978- 0-12-381479-1.00020-4

    doi:10.1016/b978- 2012

  2. [2]

    ISO/IEC/IEEE International Standard - Systems and software engineering–Vocabulary.ISO/IEC/IEEE 24765:2017(E)(2017), 1–541

    2017. ISO/IEC/IEEE International Standard - Systems and software engineering–Vocabulary.ISO/IEC/IEEE 24765:2017(E)(2017), 1–541. doi:10.1109/IEEESTD.2017.8016712

    arXiv 2017

  3. [3]

    image.no_webcam_support·Issue #10143·gradio-app/gradio

    2024. image.no_webcam_support·Issue #10143·gradio-app/gradio. https://github.com/gradio-app/gradio/issues/ 10143. [Accessed 16-12-2025]

    2024

  4. [4]

    MCP tool doesn’t show up in Claude (macOS)·Issue #12·MarkusPfundstein/mcp-gsuite

    2024. MCP tool doesn’t show up in Claude (macOS)·Issue #12·MarkusPfundstein/mcp-gsuite. https://github.com/ MarkusPfundstein/mcp-gsuite/issues/12

    2024

  5. [5]

    Architecture overview

    2025. Architecture overview. https://modelcontextprotocol.io/docs/learn/architecture. [Accessed 16-12-2025]

    2025

  6. [6]

    (Bedrock Knowledge Base Retrieval MCP): (403 Error when accessing Bedrock Knowledge Base through QueryKnowledgeBases tool)·Issue #237·awslabs/mcp

    2025. (Bedrock Knowledge Base Retrieval MCP): (403 Error when accessing Bedrock Knowledge Base through QueryKnowledgeBases tool)·Issue #237·awslabs/mcp. https://github.com/awslabs/mcp/issues/237

    2025

  7. [7]

    [BUG] Dot in Project Path Causes Errors·Issue #226·basicmachines-co/basic-memory

    2025. [BUG] Dot in Project Path Causes Errors·Issue #226·basicmachines-co/basic-memory. https://github.com/ basicmachines-co/basic-memory/issues/226

    2025

  8. [8]

    [BUG] Latest MCP docker image fails due to missing Azure-identity package·Issue #778·getzep/graphiti

    2025. [BUG] Latest MCP docker image fails due to missing Azure-identity package·Issue #778·getzep/graphiti. https://github.com/getzep/graphiti/issues/778

    2025

Show all 171 references
  1. [9]

    [BUG]: Unity Catalog Server goes directly to ’crashed’ state in Databricks Apps·Issue #28·databrickslabs/mcp

    2025. [BUG]: Unity Catalog Server goes directly to ’crashed’ state in Databricks Apps·Issue #28·databrickslabs/mcp. https://github.com/databrickslabs/mcp/issues/28

    2025

  2. [10]

    callback http servers should not run forever·Issue #4·pipeboard-co/meta-ads-mcp

    2025. callback http servers should not run forever·Issue #4·pipeboard-co/meta-ads-mcp. https://github.com/ pipeboard-co/meta-ads-mcp/issues/4

    2025

  3. [11]

    Cannot install dicom-mcp and dicom-mcp[dev]==0.1.1 because these package versions have conflicting dependencies·Issue #14·ChristianHinge/dicom-mcp

    2025. Cannot install dicom-mcp and dicom-mcp[dev]==0.1.1 because these package versions have conflicting dependencies·Issue #14·ChristianHinge/dicom-mcp. https://github.com/ChristianHinge/dicom-mcp/issues/14

    2025

  4. [12]

    Cannot run mcp tools with google-adk agents·Issue #13·Rudra-ravi/wikipedia-mcp

    2025. Cannot run mcp tools with google-adk agents·Issue #13·Rudra-ravi/wikipedia-mcp. https://github.com/Rudra- ravi/wikipedia-mcp/issues/13

    2025

  5. [13]

    Claude Code hook configuration broken - multi-agent file locking non-functional·Issue #32·rinadelph/Agent- MCP

    2025. Claude Code hook configuration broken - multi-agent file locking non-functional·Issue #32·rinadelph/Agent- MCP. https://github.com/rinadelph/Agent-MCP/issues/32

    2025

  6. [14]

    Claude completely lost it: all schemas fail·Issue #71·rusiaaman/wcgw

    2025. Claude completely lost it: all schemas fail·Issue #71·rusiaaman/wcgw. https://github.com/rusiaaman/wcgw/ issues/71

    2025

  7. [15]

    Claude Desktop can’t connect·Issue #10·Rudra-ravi/wikipedia-mcp

    2025. Claude Desktop can’t connect·Issue #10·Rudra-ravi/wikipedia-mcp. https://github.com/Rudra-ravi/wikipedia- mcp/issues/10

    2025

  8. [16]

    connection refuse·Issue #621·PDFMathTranslate/PDFMathTranslate

    2025. connection refuse·Issue #621·PDFMathTranslate/PDFMathTranslate. https://github.com/PDFMathTranslate/ PDFMathTranslate/issues/621. [Accessed 16-12-2025]

    2025

  9. [17]

    Duplicate version tool in MCP tool list·Issue #120·BeehiveInnovations/zen-mcp-server

    2025. Duplicate version tool in MCP tool list·Issue #120·BeehiveInnovations/zen-mcp-server. https://github.com/ BeehiveInnovations/zen-mcp-server/issues/120

    2025

  10. [18]

    entity_id gets "correct" to wrong string·Issue #11·voska/hass-mcp

    2025. entity_id gets "correct" to wrong string·Issue #11·voska/hass-mcp. https://github.com/voska/hass-mcp/ issues/11 J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026. Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy 111:33

    2025

  11. [19]

    ‘ERROR: Exception in ASGI application‘ when running MCP with Docker·Issue #470·getzep/graphiti

    2025. ‘ERROR: Exception in ASGI application‘ when running MCP with Docker·Issue #470·getzep/graphiti. https://github.com/getzep/graphiti/issues/470

    2025

  12. [20]

    Fatal exception: Project name ’XXXXX’ already exists and points to <some other repository>·Issue #314· oraios/serena

    2025. Fatal exception: Project name ’XXXXX’ already exists and points to <some other repository>·Issue #314· oraios/serena. https://github.com/oraios/serena/issues/314

    2025

  13. [21]

    get_conversation and list_conversations tools not exposed despite being defined in code·Issue #51· elevenlabs/elevenlabs-mcp

    2025. get_conversation and list_conversations tools not exposed despite being defined in code·Issue #51· elevenlabs/elevenlabs-mcp. https://github.com/elevenlabs/elevenlabs-mcp/issues/51

    2025

  14. [22]

    Gmail Batch API Causes Consistent SSL/TLS Crashes in Python MCP Server·Issue #115·taylorwilsdon/google_- workspace_mcp

    2025. Gmail Batch API Causes Consistent SSL/TLS Crashes in Python MCP Server·Issue #115·taylorwilsdon/google_- workspace_mcp. https://github.com/taylorwilsdon/google_workspace_mcp/issues/115

    2025

  15. [23]

    Incorrect Redis password environment variable name in the sample Claude Desktop configuration·Issue #6· redis/mcp-redis

    2025. Incorrect Redis password environment variable name in the sample Claude Desktop configuration·Issue #6· redis/mcp-redis. https://github.com/redis/mcp-redis/issues/6

    2025

  16. [24]

    Invalid JSON parsing when starting on Windows environment·Issue #12·doobidoo/mcp-memory-service

    2025. Invalid JSON parsing when starting on Windows environment·Issue #12·doobidoo/mcp-memory-service. https://github.com/doobidoo/mcp-memory-service/issues/12

    2025

  17. [25]

    MCP Server returns fact_embedding arrays causing 50x token inflation·Issue #610·getzep/graphiti

    2025. MCP Server returns fact_embedding arrays causing 50x token inflation·Issue #610·getzep/graphiti. https: //github.com/getzep/graphiti/issues/610

    2025

  18. [26]

    Multiple sessions: ‘The review seems to be looking at a different repository again‘·Issue #28· BeehiveInnovations/zen-mcp-server

    2025. Multiple sessions: ‘The review seems to be looking at a different repository again‘·Issue #28· BeehiveInnovations/zen-mcp-server. https://github.com/BeehiveInnovations/zen-mcp-server/issues/28

    2025

  19. [27]

    [OpenMemory] Cannot connect, 404 on ‘messages‘ route, wrong URL (?)·Issue #2695·mem0ai/mem0

    2025. [OpenMemory] Cannot connect, 404 on ‘messages‘ route, wrong URL (?)·Issue #2695·mem0ai/mem0. https://github.com/mem0ai/mem0/issues/2695

    2025

  20. [28]

    2025. Prompts. https://modelcontextprotocol.io/specification/2025-11-25/server/prompts. [Accessed 16-12-2025]

    2025

  21. [29]

    REDDIT_USERNAME and REDDIT_PASSWORD are not optional for readonly operations·Issue #4· Arindam200/reddit-mcp

    2025. REDDIT_USERNAME and REDDIT_PASSWORD are not optional for readonly operations·Issue #4· Arindam200/reddit-mcp. https://github.com/Arindam200/reddit-mcp/issues/4

    2025

  22. [30]

    Resources

    2025. Resources. https://modelcontextprotocol.io/specification/2025-11-25/server/resources. [Accessed 16-12-2025]

    2025

  23. [31]

    2025. SDKs. https://modelcontextprotocol.io/docs/sdk. [Accessed 16-12-2025]

    2025

  24. [32]

    Server crashes on startup due to f-string syntax error·Issue #15·elevenlabs/elevenlabs-mcp

    2025. Server crashes on startup due to f-string syntax error·Issue #15·elevenlabs/elevenlabs-mcp. https://github. com/elevenlabs/elevenlabs-mcp/issues/15

    2025

  25. [33]

    Should not do prints in console when using STDIO transport·Issue #4·Rudra-ravi/wikipedia-mcp

    2025. Should not do prints in console when using STDIO transport·Issue #4·Rudra-ravi/wikipedia-mcp. https: //github.com/Rudra-ravi/wikipedia-mcp/issues/4

    2025

  26. [34]

    Too slow when save from Claude with openmemory·Issue #3011·mem0ai/mem0

    2025. Too slow when save from Claude with openmemory·Issue #3011·mem0ai/mem0. https://github.com/mem0ai/ mem0/issues/3011

    2025

  27. [35]

    2025. Tools. https://modelcontextprotocol.io/specification/2025-11-25/server/tools. [Accessed 16-12-2025]

    2025

  28. [36]

    Unable to connect to local mcp server over sse, when using MCP Inspector·Issue #18·elevenlabs/elevenlabs-mcp

    2025. Unable to connect to local mcp server over sse, when using MCP Inspector·Issue #18·elevenlabs/elevenlabs-mcp. https://github.com/elevenlabs/elevenlabs-mcp/issues/18

    2025

  29. [37]

    Unable to use any tool due to parameter error·Issue #19·echelon-ai-labs/servicenow-mcp

    2025. Unable to use any tool due to parameter error·Issue #19·echelon-ai-labs/servicenow-mcp. https://github. com/echelon-ai-labs/servicenow-mcp/issues/19

    2025

  30. [38]

    Understanding MCP clients

    2025. Understanding MCP clients. https://modelcontextprotocol.io/docs/learn/client-concepts. [Accessed 16-12-2025]

    2025

  31. [39]

    Understanding MCP servers

    2025. Understanding MCP servers. https://modelcontextprotocol.io/docs/learn/server-concepts. [Accessed 16-12- 2025]

    2025

  32. [40]

    with claude-code the find_symbol and related mcp tools don’t return any results·Issue #153·oraios/serena

    2025. with claude-code the find_symbol and related mcp tools don’t return any results·Issue #153·oraios/serena. https://github.com/oraios/serena/issues/153

    2025

  33. [41]

    2015.Data mining: the textbook

    Charu C Aggarwal et al. 2015.Data mining: the textbook. Vol. 1. Springer

    2015

  34. [42]

    Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shepherd. 2020. Software documentation: the practitioners’ perspective. InProceedings of the acm/ieee 42nd interna- tional conference on software engineering. 590–601

    2020

  35. [43]

    Roozbeh Aghili, Heng Li, and Foutse Khomh. 2023. Studying the characteristics of AIOps projects on GitHub. Empirical Software Engineering28, 6 (2023), 143

    2023

  36. [44]

    Khairul Alam and Banani Roy. 2026. Analyzing GitHub Issues and Pull Requests in nf-core Pipelines: Insights into nf-core Pipeline Repositories.arXiv preprint arXiv:2601.09612(2026)

    2026

  37. [45]

    Khairul Alam, Banani Roy, Chanchal K Roy, and Kartik Mittal. 2025. An empirical investigation on the challenges in scientific workflow systems development.Empirical Software Engineering30, 5 (2025), 151

    2025

  38. [46]

    Mohammad Aldhaen. 2020. Interview versus questionnaire from the perspective of CBE members.International Journal of Education, Learning and Development8, 2 (2020), 21–41

    2020

  39. [47]

    2024.Introducing the Model Context Protocol

    Anthropic. 2024.Introducing the Model Context Protocol. https://www.anthropic.com/news/model-context-protocol

    2024

  40. [48]

    Armstrong

    Richard A. Armstrong. 2014. When to use the Bonferroni correction.Ophthalmic and Physiological Optics34, 5 (2014), 502–508. doi:10.1111/opo.12131

    2014 doi

  41. [49]

    Azure-Samples. 2025. Azure-Samples/semantic-kernel-advanced-usage. https://github.com/Azure-Samples/semantic- kernel-advanced-usage J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:34 Taraghi et al

    2025

  42. [50]

    Amiangshu Bosu and Jeffrey C Carver. 2014. Impact of developer reputation on code review outcomes in oss projects: An empirical investigation. InProceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement. 1–10

    2014

  43. [51]

    Morakot Choetkiertikul, Peerachai Banyongrakkul, Chaiyong Ragkhitwetsagul, Suppawong Tuarob, Hoa Khanh Dam, and Thanwadee Sunetnanta. 2024. Sprint2Vec: a deep characterization of sprints in iterative software development. IEEE Transactions on Software Engineering51, 1 (2024), 220–242

    2024

  44. [52]

    2013.Statistical Power Analysis for the Behavioral Sciences(2 ed.)

    Jacob Cohen. 2013.Statistical Power Analysis for the Behavioral Sciences(2 ed.). Routledge, New York. doi:10.4324/ 9780203771587

    2013

  45. [53]

    GitHub developer guideline documentation. 2021. GitHub REST API. https://developer.github.com/v3/. Accessed: 2025-09-05

    2021

  46. [54]

    Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. 2025. A survey on code generation with llm-based agents.arXiv preprint arXiv:2508.00083(2025)

    2025 arXiv

  47. [55]

    Olive Jean Dunn. 1964. Multiple Comparisons Using Rank Sums.Technometrics6, 3 (Aug. 1964), 241–252. doi:10.1080/00401706.1964.10490181 Publisher: ASA Website _eprint: https://www.tandfonline.com/doi/pdf/10.1080/00401706.1964.10490181

    1964

  48. [56]

    Abul Ehtesham, Aditi Singh, Gaurav Kumar Gupta, and Saket Kumar. 2025. A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP). doi:10.48550/arXiv.2505.02279...

    2025 doi arXiv

  49. [57]

    Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman S...

    2024

  50. [58]

    Rosa Falotico and Piero Quatto. 2015. Fleiss’ kappa statistic without paradoxes.Quality & Quantity49, 2 (2015), 463–470

    2015

  51. [59]

    Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. 2024. Drive like a human: Rethinking autonomous driving with large language models. In2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (W ACVW). IEEE, 910–919

    2024

  52. [60]

    2004.Software quality assurance: from theory to implementation

    Daniel Galin. 2004.Software quality assurance: from theory to implementation. Pearson education

    2004

  53. [61]

    Ahmad Nauman Ghazi, Kai Petersen, Sri Sai Vijay Raj Reddy, and Harini Nekkanti. 2018. Survey research in software engineering: Problems and mitigation strategies.IEEE Access7 (2018), 24703–24718

    2018

  54. [62]

    Google. [n. d.]. Embeddings | Gemini API. https://ai.google.dev/gemini-api/docs/embeddings

  55. [63]

    Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. doi:10.48550/ arXiv.2203.05794 arXiv:2203.05794 [cs]

    2022 doi arXiv

  56. [64]

    Maarten Grootendorst. 2025. MaartenGr/BERTopic. https://github.com/MaartenGr/BERTopic original-date: 2020-09- 22T14:19:29Z

    2025

  57. [65]

    JSON-RPC Working Group. 2010. JSON-RPC 2.0 Specification. https://www.jsonrpc.org/specification. [Accessed 16-12-2025]

    2010

  58. [66]

    Hechuan Guo, Yongle Hao, Yue Zhang, Minghui Xu, Peizhuo Lv, Jiezhi Chen, and Xiuzhen Cheng. 2025. A Measurement Study of Model Context Protocol Ecosystem. doi:10.48550/arXiv.2509.25292 arXiv:2509.25292 [cs]

    2025 doi

  59. [67]

    Zikang Guo, Benfeng Xu, Chiwei Zhu, Wentao Hong, Xiaorui Wang, and Zhendong Mao. 2025. MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools. doi:10.48550/arXiv.2509.09734 arXiv:2509.09734 [cs]

    2025 doi arXiv

  60. [68]

    Duc Minh Ha, Phu Trac Kien, Tho Quan, and Anh Nguyen-Duc. 2025. Evaluating Classical Software Process Models as Coordination Mechanisms for LLM-Based Software Generation.arXiv preprint arXiv:2509.13942(2025)

    2025

  61. [69]

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al . 2023. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects.Authore...

    2023

  62. [70]

    Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2025. Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers. arXiv preprint arXiv:2506.13538(2025)

    2025 arXiv

  63. [71]

    HazyResearch. 2026. HazyResearch/minions. https://github.com/HazyResearch/minions

    2026

  64. [72]

    Kim Herzig, Sascha Just, and Andreas Zeller. 2013. It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In2013 35th international conference on software engineering (ICSE). IEEE, 392–401

    2013

  65. [73]

    Maria Cristina Hinojosa Lee, Johan Braet, and Johan Springael. 2024. Performance metrics for multilabel emotion classification: comparing micro, macro, and weighted f1-scores.Applied Sciences14, 21 (2024), 9863

    2024

  66. [74]

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278(2025)

    2025 arXiv

  67. [75]

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions. doi:10.48550/arXiv.2503.23278 arXiv:2503.23278 [cs]

    2025 doi arXiv

  68. [76]

    Hugging Face. 2024. gemma-7b. https://huggingface.co/google/gemma-7b. Accessed: 10/09/2025

    2024

  69. [77]

    Hugging Face. 2024. Llama-3.1-8B. https://huggingface.co/meta-llama/Llama-3.1-8B. Accessed: 10/09/2025

    2024

  70. [78]

    Hugging Face. 2024. zephyr-7b. https://huggingface.co/HuggingFaceH4/zephyr-7b-beta. Accessed: 10/09/2025

    2024

  71. [79]

    Hugging Face. 2025. Mistral-7B-v0.1. https://huggingface.co/mistralai/Mistral-7B-v0.1. Accessed: 10/09/2025

    2025

  72. [80]

    Hugging Face. 2025. Qwen3-8B. https://huggingface.co/Qwen/Qwen3-8B. Accessed: 10/09/2025

    2025

  73. [81]

    HuggingFace. [n. d.]. MTEB Leaderboard - a Hugging Face Space by mteb. https://huggingface.co/spaces/mteb/ leaderboard

  74. [82]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. InProceedings of the ACM/IEEE 42nd international conference on software engineering. 1110–1121

    2020

  75. [83]

    Quality Assessment Instruments. 2012. Validity and inter-rater reliability testing of quality assessment instruments. (2012)

    2012

  76. [84]

    Niful Islam, Ragib Shahriar Ayon, Deepak George Thomas, Shibbir Ahmed, and Mohammad Wardat. 2026. When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labeling. doi:10.48550/arXiv.2601.15232 arXiv:2601.15232 [cs]

    2026 doi arXiv

  77. [85]

    Weipeng Jiang, Xiaoyu Zhang, Xiaofei Xie, Jiongchi Yu, Yuhan Zhi, Shiqing Ma, and Chao Shen. 2025. The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries. https://arxiv.org/abs/2506.12320v1

    2025 arXiv

  78. [86]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models.arXiv preprint arXiv:2307.10169(2023)

    2023 arXiv

  79. [87]

    Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2021. Predicting issue types on GitHub. Science of Computer Programming205 (2021), 102598

    2021

  80. [88]

    Vigdis By Kampenes, Tore Dybå, Jo E Hannay, and Dag IK Sjøberg. 2007. A systematic review of effect size in software engineering experiments.Information and Software Technology49, 11-12 (2007), 1073–1086

    2007

  81. [89]

    Oleksii Kononenko, Tresa Rose, Olga Baysal, Michael Godfrey, Dennis Theisen, and Bart De Water. 2018. Studying pull request merges: A case study of shopify’s active merchant. InProceedings of the 40th international conference on software engineering: software engineering in pr...

    2018

  82. [90]

    kotaro.kinoshita. 2026. kotaro-kinoshita/yomitoku. https://github.com/kotaro-kinoshita/yomitoku

    2026

  83. [91]

    Paul Krill. 2025. Python popularity climbs to highest ever. https://www.infoworld.com/article/3981643. [Accessed 16-12-2025]

    2025

  84. [92]

    Kruskal and W

    William H. Kruskal and W. Allen Wallis. 1952. Use of Ranks in One-Criterion Variance Analysis.J. Amer. Statist. Assoc.47, 260 (Dec. 1952), 583–621. doi:10.1080/01621459.1952.10483441 Publisher: ASA Website _eprint: https://www.tandfonline.com/doi/pdf/10.1080/01621459.1952.10483441

    1952

  85. [93]

    Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, ...

    2025 doi arXiv

  86. [94]

    Heng Li, Foutse Khomh, Moses Openja, et al. 2021. Understanding quantum software engineering challenges an empirical study on stack exchange forums and github issues. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 343–354

    2021

  87. [95]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation.arXiv preprint arXiv:2101.00190(2021). J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:36 Taraghi et al

    2021 arXiv

  88. [96]

    Yun Li, Kai Katsumata, Ehsan Javanmardi, and Manabu Tsukada. 2024. Large language models for human-like autonomous driving: A survey. In2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 439–446

    2024

  89. [97]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large language models in finance: A survey. InProceedings of the fourth ACM international conference on AI in finance. 374–382

    2023

  90. [98]

    Zhixing Li, Yue Yu, Tao Wang, Yan Lei, Ying Wang, and Huaimin Wang. 2022. To follow or not to follow: Understanding issue/pull-request templates on github.IEEE Transactions on Software Engineering49, 4 (2022), 2530–2544

    2022

  91. [99]

    LiteLLM. [n. d.].LiteLLM: Call all LLM APIs using the OpenAI format. https://github.com/BerriAI/litellm/issues/6600

  92. [100]

    Mugeng Liu, Siqi Zhong, Weichen Bi, Yixuan Zhang, Zhiyang Chen, Zhenpeng Chen, Xuanzhe Liu, and Yun Ma. 2026. A First Look at Bugs in LLM Inference Engines.ACM Trans. Softw. Eng. Methodol.(Jan. 2026). doi:10.1145/3788873 Just Accepted

    2026 doi

  93. [101]

    Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. 2024. Datasets for large language models: A comprehensive survey.arXiv preprint arXiv:2402.18041(2024)

    2024 arXiv

  94. [102]

    Guoming Long, Jingzhi Gong, Hui Fang, and Tao Chen. 2025. Learning software bug reports: a systematic literature review.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  95. [103]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.arXiv preprint arXiv:2104.08786(2021)

    2021 arXiv

  96. [104]

    Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah, Pradeep Honaganahalli Basavaraju, and James A. Burke. 2026. ScaleMCP: Dynamic and Auto-synchronizing Model Context Protocol Tools for LLM Agents. InComputational Intel- ligence, Francesco Marcelloni, Kurosh Madani, Niki van Stein...

    2026 doi

  97. [105]

    Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. 2025. MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers. doi:10.48550/arXiv.2508.1470...

    2025 doi arXiv

  98. [106]

    H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other.The Annals of Mathematical Statistics18, 1 (1947), 50–60. https://www.jstor.org/stable/2236101 Publisher: Institute of Mathematical Statistics

    1947

  99. [107]

    John McCarthy et al. 2007. What is artificial intelligence. (2007)

    2007

  100. [108]

    Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering.Journal of Open Source Software2, 11 (March 2017), 205. doi:10.21105/joss.00205

    2017 doi

  101. [109]

    Leland McInnes, John Healy, and James Melville. 2020. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. doi:10.48550/arXiv.1802.03426 arXiv:1802.03426 [stat]

    2020 doi arXiv

  102. [110]

    McKight and Julius Najab

    Patrick E. McKight and Julius Najab. 2010. Kruskal-Wallis Test. InThe Corsini Encyclope- dia of Psychology. John Wiley & Sons, Ltd, 1–1. doi:10.1002/9780470479216.corpsy0491 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470479216.corpsy0491

    2010 doi

  103. [111]

    Raphael Meier. 2024. Llm-aided social media influence operations.Large Language Models in Cybersecurity: Threats, Exposure and Mitigation(2024), 105–112

    2024

  104. [112]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey.arXiv preprint arXiv:2402.06196(2024)

    2024 arXiv

  105. [113]

    Guozhao Mo, Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun

  106. [114]

    Mohammad Mehdi Morovati, Amin Nikanjam, and Foutse Khomh. 2024. Fault Localization in Deep Learning-based Software: A System-level Approach.arXiv preprint arXiv:2411.08172(2024)

    2024 arXiv

  107. [115]

    Mohammad Mehdi Morovati, Amin Nikanjam, Foutse Khomh, and Zhen Ming Jiang. 2023. Bugs in machine learning- based systems: a faultload benchmark.Empirical Software Engineering28, 3 (2023), 62

    2023

  108. [116]

    Mohammad Mehdi Morovati, Amin Nikanjam, Florian Tambon, Foutse Khomh, and Zhen Ming Jiang. 2024. Bug characterization in machine learning-based systems.Empirical Software Engineering29, 1 (2024), 14

    2024

  109. [117]

    Mohammad Mehdi Morovati, Florian Tambon, Mina Taraghi, Amin Nikanjam, and Foutse Khomh. 2024. Com- mon challenges of deep reinforcement learning applications development: an empirical study.Empirical Software Engineering29, 4 (2024), 95

    2024

  110. [118]

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2025. A comprehensive overview of large language models.ACM Transactions on Intelligent Systems and Technology16, 5 (2025), 1–72

    2025

  111. [119]

    2016.Surveys in software engineering: a systematic literature review and interview study

    Harini Nekkanti and SSVR Reddy. 2016.Surveys in software engineering: a systematic literature review and interview study. Ph. D. Dissertation. MSc thesis, Blekinge Institute of Technology, Sweden

    2016

  112. [120]

    NirDiamant. 2026. NirDiamant/GenAI_Agents. https://github.com/NirDiamant/GenAI_Agents J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026. Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy 111:37

    2026

  113. [121]

    2024.Python Regular Expression

    Python official documentation. 2024.Python Regular Expression. https://docs.python.org/3/library/re.html

    2024

  114. [122]

    Ollama. 2024. gemma3:4b. https://ollama.com/library/gemma3:4b

    2024

  115. [123]

    Ollama. 2024. llama3.1. https://ollama.com/library/llama3.1. Accessed: 10/09/2025

    2024

  116. [124]

    Ollama. 2024. mistral:7b. https://ollama.com/library/mistral:7b

    2024

  117. [125]

    Ollama. 2024. Ollama. https://github.com/ollama/ollama. Accessed: 10/09/2025

    2024

  118. [126]

    Ollama. 2024. qwen3:8b. https://ollama.com/library/qwen3:8b

    2024

  119. [127]

    Ollama. 2024. zephyr:7b. https://ollama.com/library/zephyr

    2024

  120. [128]

    OpenAI. 2025. The official Python library for the OpenAI API. https://github.com/openai/openai-python

    2025

  121. [129]

    Paper authors. 2025. Paper replication package. https://github.com/mcpbugtaxonomy/mcp-bug-taxonomy. Accessed: 01/10/2025

    2025

  122. [130]

    Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python.the Journal of machine Learning research12 (2011), 2825–2830

    2011

  123. [131]

    Qualtrics. 2025. Qualtrics official website. https://www.qualtrics.com/. Accessed: 2025-11-01

    2025

  124. [132]

    Sandra L Ramírez-Mora, Hanna Oktaba, and Helena Gómez-Adorno. 2020. Descriptions of issues and comments for predicting issue success in software projects.Journal of Systems and Software168 (2020), 110663

    2020

  125. [133]

    Partha Pratim Ray. 2025. A Review on Agent-to-Agent Protocol: Concept, State-of-the-art, Challenges and Future Directions. doi:10.36227/techrxiv.174612014.42157096/v1

    2025

  126. [134]

    John TE Richardson. 2011. Eta squared and partial eta squared as measures of effect size in educational research. Educational research review6, 2 (2011), 135–147

    2011

  127. [135]

    Alan Romano, Xinyue Liu, Yonghwi Kwon, and Weihang Wang. 2021. An empirical study of bugs in webassembly compilers. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 42–54

    2021

  128. [136]

    2021.The Coding Manual for Qualitative Researchers

    Johnny Saldana. 2021.The Coding Manual for Qualitative Researchers. SAGE Publications Inc., Thousand Oaks, CA

    2021

  129. [137]

    Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering.IEEE Transactions on software engineering25, 4 (1999), 557–572

    1999

  130. [138]

    Sheona Shackleton, Joana Carlos Bezerra, Jessica Cockburn, Maureen G Reed, and Razak Abu. 2021. Interviews and surveys. InThe Routledge handbook of research methods for social-ecological systems. Routledge, 107–118

    2021

  131. [139]

    Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika52, 3/4 (1965), 591–611. doi:10.2307/2333709

    1965 doi

  132. [140]

    Sina Shool, Sara Adimi, Reza Saboori Amleshi, Ehsan Bitaraf, Reza Golpira, and Mahmood Tara. 2025. A systematic review of large language model (LLM) evaluations in clinical medicine.BMC Medical Informatics and Decision Making 25, 1 (2025), 117

    2025

  133. [141]

    Mohammed Latif Siddiq and Joanna CS Santos. 2022. Bert-based github issue report classification. InProceedings of the 1st international workshop on natural language-based software engineering. 33–36

    2022

  134. [142]

    Kristina P Sinaga and Miin-Shen Yang. 2020. Unsupervised K-means clustering algorithm.IEEE access8 (2020), 80716–80727

    2020

  135. [143]

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. A survey of the model context protocol (mcp): Standardizing context to enhance large language models (llms). (2025)

    2025

  136. [144]

    Mozhan Soltani, Felienne Hermans, and Thomas Bäck. 2020. The significance of bug report elements.Empirical Software Engineering25, 6 (2020), 5255–5294

    2020

  137. [145]

    Wei Song, Haonan Zhong, Ziqi Ding, Jingling Xue, and Yuekang Li. 2025. Help or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models. doi:10.48550/arXiv.2508.12566 arXiv:2508.12566 [cs]

    2025 doi arXiv

  138. [146]

    Llama Stack. 2026. llamastack/llama-stack. https://github.com/llamastack/llama-stack

    2026

  139. [147]

    Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2025. Bugs in large language models generated code: An empirical study.Empirical Software Engineering 30, 3 (2025), 65

    2025

  140. [148]

    Maksim Terpilowski. 2019. scikit-posthocs: Pairwise multiple comparison tests in Python.Journal of Open Source Software4, 36 (April 2019), 1169. doi:10.21105/joss.01169

    2019 doi

  141. [149]

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine.Nature medicine29, 8 (2023), 1930–1940

    2023

  142. [150]

    Haoye Tian, Chong Wang, BoYang Yang, Lyuye Zhang, and Yang Liu. 2025. A Taxonomy of Prompt Defects in LLM Systems. https://arxiv.org/abs/2509.14404v1

    2025

  143. [151]

    2005.Software quality engineering: testing, quality assurance, and quantifiable improvement

    Jeff Tian. 2005.Software quality engineering: testing, quality assurance, and quantifiable improvement. John Wiley & Sons

    2005

  144. [152]

    Maciej Tomczak and Ewa Tomczak. 2014. The need to report effect size estimates revisited. An overview of some recommended measures of effect size. https://www.wbc.poznan.pl/dlibra/publication/413565 Publication Title: Trends in Sport Sciences 2014 Vol.21 No.1. J. ACM, Vol. 37,...

    2014

  145. [153]

    Lukas Twist, Jie M Zhang, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, and Detlef Nauck. 2025. A study of llms’ preferences for libraries and programming languages.arXiv preprint arXiv:2503.17181(2025)

    2025 arXiv

  146. [154]

    András Vargha and Harold D Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong.Journal of Educational and Behavioral Statistics25, 2 (2000), 101–132

    2000

  147. [155]

    Lev Velykoivanenko, Kavous Salehzadeh Niksirat, Stefan Teofanovic, Bertil Chapuis, Michelle L Mazurek, and Kévin Huguenin. 2024. Designing a data-driven survey system: leveraging participants’ online data to personalize surveys. InProceedings of the 2024 CHI Conference on Huma...

    2024

  148. [156]

    Giri Vijayaraghavan and Cem Kaner. 2003. Bug taxonomies: Use them to generate better tests.Star East2003 (2003), 1–40

    2003

  149. [157]

    Vaishali Vinay. 2025. Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications. doi:10.48550/arXiv.2511.19933 arXiv:2511.19933 [cs]

    2025 doi

  150. [158]

    Vaishali Vinay. 2025. A System-Level Taxonomy of Failure Modes in Large Language Model Applications.arXiv preprint arXiv:2511.19933(2025)

    2025

  151. [159]

    Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python.Nature methods17, 3 (2020), 261–272

    2020

  152. [160]

    Huanting Wang, Jingzhi Gong, Huawei Zhang, Jie Xu, and Zheng Wang. 2025. AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities. doi:10.48550/arXiv.2508.11126 arXiv:2508.11126 [cs]

    2025 doi

  153. [161]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682 (2022)

    2022 arXiv

  154. [162]

    Cailin Winston and René Just. 2025. A taxonomy of failures in tool-augmented llms. In2025 IEEE/ACM International Conference on Automation of Software Test (AST). IEEE, 125–135

    2025

  155. [163]

    Cailin Winston and René Just. 2025. A Taxonomy of Failures in Tool-Augmented LLMs. In2025 IEEE/ACM International Conference on Automation of Software Test (AST). 125–135. doi:10.1109/AST66626.2025.00019 ISSN: 2833-9061

    2025

  156. [164]

    Ziluo Xue, Yanjie Zhao, Shenao Wang, Kai Chen, and Haoyu Wang. 2025. A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 3369–3380. doi:10.1109/ASE63991.2025.002...

    2025

  157. [165]

    Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong, Mian Zhang, Yebowen Hu, Shujian Liu, Simin Ma, Song Wang, Sathish Reddy Indurthi, Xun Wang, Yiran Chen, and Kaiqiang Song. 2025. LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries. do...

    2025 doi arXiv

  158. [166]

    Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, et al. 2024. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making.Advances in Neura...

    2024

  159. [167]

    yzfly. 2026. yzfly/douyin-mcp-server. https://github.com/yzfly/douyin-mcp-server

    2026

  160. [168]

    role": "system

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. InInternational conference on machine learning. PMLR, 12697–12706. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026...

    2021

  161. [170]

    Provide a concise SUBJECT label that summarizes the actual issue, like a bug category or specific technical error (e.g.,'NullPointerException in ConfigLoader','Docker build fails on Alpine')

  162. [171]

    Respond in the following format: Subject: <subject here> Summary: <summary here> """} Listing 3

    Write a 1-2 sentence SUMMARY explaining what the issue was and what caused or resolved it. Respond in the following format: Subject: <subject here> Summary: <summary here> """} Listing 3. Prompt template for issue summarization J. ACM, Vol. 37, No. 4, Article 111. Publication ...

    2026

  163. [2025]

    LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? doi:10.48550/arXiv.2508.01780 arXiv:2508.01780 [cs]

    doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.