{"id":"fbd0db3a-e0e9-4309-ab74-169212638150","arxiv_id":"2603.05637","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MCP server faults form five empirical categories—server setting, server/tool configuration, server/host configuration, documentation, and general programming—confirmed by a 41-practitioner survey.","lead":"The paper mines 26,821 closed GitHub issues from 385 Python-based MCP server repositories, hand-codes 407 MCP-related bugs, and organizes them into five fault categories. A 41-respondent practitioner survey reports that all categories occur in practice, while statistics show MCP faults differ from non-MCP faults in discussion volume and fixer experience.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 407-issue sample may be a biased convenience sample: the MCP-relatedness filter's recall is unmeasured, so 'comprehensive' is not yet supported.","rationale":"The reader's CONDITIONAL verdict is appropriate. The most load-bearing assumption is that the 407-issue corpus is a representative sample of MCP-server faults; the reader identified this same region, though with a compound concern (LLM triage, cluster review, and no open-coding agreement). I focus narrowly on the recall of the MCP-relatedness filter because low recall directly undermines the 'comprehensive' claim: missing a class of MCP faults cannot be corrected by any amount of manual labeling of the included issues. The open-coding agreement issue is real but secondary; it could be checked by re-annotating the 407 issues and would affect category reliability, not sample representativeness. The small survey (41 respondents) is a weak external validation, but it was designed only to confirm categories already derived, so it cannot detect omitted categories. The paper deserves credit for manual labeling of all 407 issues, reporting prompt templates, and providing a replication package, and the taxonomy may well be useful as a first-step characterization of MCP-tagged issue reports. But the word 'comprehensive' outruns the evidence until the recall of the selection pipeline is quantified. The proposed test directly measures that recall. If it shows substantial missed MCP faults, the authors should either apply a more sensitive filter or revise the claim to 'a taxonomy of MCP-related issues explicitly discussed as such.'","tokens_in":37960,"tokens_out":5523,"duration_ms":56815,"concrete_test":"Randomly sample 384 of the 2,875 bug issues that were excluded from the final 407 MCP-related set (§3.2.2, Table 2). Have two MCP-experienced annotators independently label each excluded issue as 'MCP-server fault' or 'not an MCP-server fault' using the paper's codebook definitions; compute Cohen's kappa and the estimated proportion of missed MCP faults with a 95% CI. If the CI lower bound exceeds ~2–3%, the recall of the filtering pipeline is too low to support 'comprehensive'; if it is near zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a 'comprehensive' taxonomy depends on the 407 closed issues in §3.2.2 being a representative sample of real MCP-server faults. The filtering chain has no measured recall. After LLM-based bug classification, BERTopic clusters the 3,282 bug issues into 101 clusters; each cluster is judged MCP-related or not by manually reviewing only 20 representative issues (§3.2.2). Issues in clusters labeled non-MCP are recovered only if their LLM-generated summary contains the literal token 'MCP' (keyword matching). This has two failure modes: (1) MCP-specific bugs whose summaries do not contain 'MCP' and that fall into mostly non-MCP clusters are silently dropped; (2) a cluster containing a substantial minority of MCP issues can be labeled non-MCP if its 20 sampled issues are dominated by non-MCP content. Because issue titles/bodies in MCP-server repos often omit the acronym (e.g., 'server crashes on startup' in an MCP server repo), false negatives are plausible. The 20-issue manual review estimates cluster-level precision, not recall, and the keyword fallback only catches explicitly labeled issues. The earlier bug/non-bug classifier (Table 1), selected on 40 hand-labeled issues with macro F1=0.77, can also drop MCP bugs if they are misclassified as questions or feature requests before clustering. The result is that the five categories and their subcategory proportions are derived from a convenience sample of issues that are explicitly MCP-tagged or cluster with obvious MCP topics, not from all MCP-server faults. This directly undermines the 'comprehensive' claim and the survey-based validation, since the survey asks about categories already extracted from a possibly biased sample and cannot detect a fault type absent from the 407 issues.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of faults in Model Context Protocol (MCP) servers. The authors mine 385 GitHub repositories that use the Python MCP SDK, extract 26,821 closed issues, classify them with an LLM (GPT-4o-mini) into bug and non-bug categories, cluster the 3,282 bug issues with BERTopic, and manually identify 407 MCP-related issues. These issues are open-coded to produce a five-category taxonomy: Server Setting, Server/Tool Configuration, Server/Host Configuration, Documentation, and General Programming, with a detailed subcategory structure. The taxonomy is said to be validated by a survey of 41 MCP practitioners, and the paper further compares MCP-related versus non-MCP faults on metrics such as fix time, number of comments, collaborators, and fixer experience. The central claim is that this is the first large-scale, comprehensive taxonomy of real faults in MCP servers, and that all identified categories occur in practice.","tokens_in":38340,"tokens_out":6224,"duration_ms":69428,"significance":"If the taxonomy is sound, this is a valuable contribution to the emerging area of MCP reliability engineering. The paper has clear strengths: the methodology is transparently described, the taxonomy is data-derived rather than imposed a priori, concrete issue examples ground each category, a replication package is promised, and the survey explicitly involves practitioners who did not construct the taxonomy. The statistical analyses report effect sizes, not just p-values, which is good practice. However, the degree to which the taxonomy is 'comprehensive' depends on the recall of the automated filtering pipeline, which is not measured. The small survey and its recruitment from the same repository population limit the strength of the validation. These concerns do not invalidate the taxonomy as a useful initial structure, but they do affect the strength of the central claim and the generalizability of the quantitative proportions.","major_comments":[{"comment":"The MCP-relatedness filter has no measured recall. Clusters are labeled MCP-related by manually reviewing 20 representative issues per cluster, and issues in non-MCP clusters are recovered only if their LLM-generated summary contains the literal token 'MCP'. Because issue titles and summaries often omit the acronym (e.g., 'server crashes on startup' in an MCP-server repository), MCP-specific issues assigned to non-MCP clusters are silently dropped. This directly affects the 407-issue sample, the taxonomy structure, and the category proportions in Fig. 3. The paper's threat-to-validity discussion acknowledges this but does not quantify the risk. Please add a recall estimate (e.g., manually inspect a random sample from non-MCP clusters or from issues whose summaries do not contain 'MCP') or explicitly soften the 'comprehensive' claim to one about the population of issues that are explicitl","section":"§3.2.2 and §6.2"},{"comment":"The statistical comparison of fault characteristics treats labels as independent units. The paper reports 419 labels from 383 issues after excluding 24 issues, stating that some issues received multiple labels. For metrics such as 'required time to fix,' an issue with two labels contributes to both groups in the Kruskal–Wallis and Dunn tests, violating the independence assumption and potentially inflating significance or distorting effect sizes. Please use a primary label per issue for the RQ3 analysis, or otherwise account for the multi-label structure. This is important because the significant differences reported in Table 6 may not be robust.","section":"§4.1 and §4.3 (Tables 5 and 6)"},{"comment":"The open-coding step is the sole basis for the taxonomy, but no inter-rater reliability statistic is reported. The paper states this is because no predefined labels were available. While this is understandable for emergent coding, the central claim would be substantially strengthened by reporting the number of disagreements resolved, a post-hoc agreement check on a random subset using the final codebook, or a more detailed justification of why a reliability measure is impossible. Without this, readers cannot distinguish robust categories from the systematic bias of the two raters. The current description of regular meetings and third-adjudicator resolution is helpful, but it does not quantify reliability.","section":"§3.2.3"},{"comment":"The survey is presented as validating the completeness of the taxonomy, but the evidence is weak. There are 41 valid responses, recruited from contributors to the same repositories that produced the issue sample; the response rate is not reported. Table 4 shows that every category is encountered by at least some respondents, but this only confirms that each category is non-empty; it does not establish completeness or external generalizability. The absence of newly suggested fault types from 41 respondents is not strong evidence of coverage. Please reframe the survey as a preliminary qualitative check, report recruitment and response-rate information, and avoid the conclusion that it 'supports the completeness' of the taxonomy.","section":"§3.3 and §4.2 (Table 4)"}],"minor_comments":[{"comment":"The conclusion says '443 MCP server repositories,' while §3.1 reports 385 repositories after manual checking. This inconsistency should be fixed.","section":"§7 (Conclusion)"},{"comment":"The relationship between 407 inspected issues, 24 excluded issues, and 419 labels should be stated more clearly. The current text says 'Each issue was examined and labeled... resulting in 419 labels' and then mentions 24 exclusions; readers may wonder whether the percentages are computed on labels or issues.","section":"§4.1"},{"comment":"The methodology figure contains the typo 'MPC-based Repos'; it should read 'MCP-based Repos'.","section":"Figure 2"},{"comment":"The table reports prevalence percentages without giving the denominator for each row. Since the number of respondents who answered each category group may differ, include N per row or clarify how missing responses are handled.","section":"Table 4"},{"comment":"The paper says HDBSCAN was used to estimate the number of clusters and then KMeans was applied with k=101. It is not clear how the HDBSCAN result maps to 101; a brief description or a sensitivity check would improve reproducibility.","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This paper addresses a timely and under-studied topic, and the taxonomy is likely to be useful to the MCP software engineering community. The main issue is the gap between the 'comprehensive' claim and the unmeasured recall of the automated pipeline that selects the 407 issues. The survey, while a good idea, is too small and too closely tied to the source population to independently validate completeness. I believe the paper can be revised to an acceptable level by adding a recall-oriented validation or by softening the claims, along with fixing the multi-label independence problem in RQ3. The work is within the scope of this journal and does not suffer from any apparent circularity or fabricated entities."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives the field its first empirically grounded taxonomy of faults in MCP servers. That is real value: the five high-level categories and their subcategories are plausible, well-illustrated with concrete issues, and the manual labeling of 407 issues is a genuine effort. The survey of 41 practitioners adds some external confirmation, and the comparison of MCP vs non-MCP fault characteristics is a nice bonus. I would not have guessed that tool response handling would be the most frequently encountered fault type, so the data carries signal.\n\nThe weak spot is the sampling pipeline, and it lands on the word \"comprehensive.\" The chain that gets from 26,821 closed issues to 407 MCP-related ones has an unmeasured recall. The LLM bug classifier was validated on 40 issues with macro F1≈0.77. The MCP-relatedness filter examines only 20 issues per cluster, and the fallback for non-MCP clusters is keyword matching on LLM-generated summaries that contain the literal token \"MCP.\" Any MCP bug whose summary omits that token and whose cluster is judged non-MCP is silently dropped. That means the taxonomy could be missing whole fault categories, and the category proportions could be off. The survey cannot fix this: participants were recruited from the same repository population that produced the sample, so they can only confirm categories already present. This is a real limitation, though an addressable one.\n\nThere are also smaller issues: the paper reports 385 repositories in Section 3.1 but says 443 in the conclusion, and the open-coding step reports no inter-rater agreement (the paper says this is because no predefined labels existed, but Fleiss' kappa was computed for cluster labeling, so a similar approach might have been possible). These are fixable in revision.\n\nI would not desk-reject this. For a protocol adopted as quickly as MCP, a first taxonomy of real faults is a useful reference for both practitioners and test-generation researchers. The right move is to send it to peer review with a request to either measure the recall of the filtering pipeline on a larger ground-truth set or explicitly scope the claims to \"a taxonomy of faults visible in this sample.\" The contribution survives that revision; the word \"comprehensive\" probably does not.","headline":"Useful first taxonomy of MCP server faults, but the 'comprehensive' framing outruns the evidence; still worth a serious referee.","tokens_in":38871,"tokens_out":1584,"would_cite":true,"duration_ms":20450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that real-world MCP server faults cluster into five empirically derived categories, and a practitioner survey confirms all five occur in practice.","keywords":["Model Context Protocol","MCP servers","software fault taxonomy","LLM-based systems","empirical software engineering","issue mining","tool integration","configuration faults"],"falsifier":"Label a fresh random sample of 200 closed bug issues from MCP server projects written in non-Python SDKs using the paper's codebook. If a substantial fraction (say more than 15-20%) of MCP-related bugs fall outside the five categories, or if the category distribution shifts dramatically, the taxonomy's completeness and generalizability claims fail.","tokens_in":37858,"feed_emoji":"🐛","tokens_out":3820,"duration_ms":38464,"temperature":0.7,"pith_summary":"This study tries to establish that the faults that actually break Model Context Protocol (MCP) servers are not random or idiosyncratic: they cluster into five recognizable categories. Analyzing 407 bug-related closed issues from 385 open-source MCP server projects, the authors derive a taxonomy whose largest categories are server/tool configuration, server/host configuration, and server setting, followed by documentation and general programming. A survey of 41 MCP practitioners reports encountering every category in practice, which the authors take as evidence the taxonomy is complete. If right, the taxonomy gives developers a checklist of where MCP systems fail and gives researchers a baseline for MCP-specific testing and repair tools.","feed_headline":"Real MCP server faults fall into five categories, study finds","feed_subtitle":"Analysis of 407 closed issues plus a practitioner survey maps where MCP servers break—mostly at configuration seams.","key_machinery":"The load-bearing object is the taxonomy itself, anchored in MCP's host-client-server architecture. Faults are classified according to where in the MCP stack the break happens: in the server's own setting, in how the server wires tools, in how the host connects to the server, in documentation, or in ordinary programming errors. The taxonomy works by taking issue reports and assigning each to a category based on the component boundary where the fault manifests, which is what lets the study compare fault types and separate MCP-specific problems from generic ones.","core_discovery":"The central claim is that the first empirically grounded taxonomy of MCP server faults consists of five high-level categories: Server Setting (27.45% of labeled issues), Server/Tool Configuration (31.74%), Server/Host Configuration (28.64%), Documentation (6.92%), and General Programming (5.25%). The study also finds that MCP-specific faults differ measurably from non-MCP faults: they attract more comments and more comments per collaborator, yet are fixed faster and by developers with less overall experience. The authors interpret the survey results as confirming that all taxonomy categories occur in practice, with tool response handling and documentation reported most frequently and tool di","pith_inferences":["The taxonomy's three dominant categories are all configuration seams, suggesting that MCP's fragility lives at integration boundaries rather than in the core protocol logic; a testable consequence is that configuration-focused static analysis would catch most avoidable failures.","The survey's finding that tool discovery/registration is rare but rated critical implies that availability failures outweigh frequency when prioritizing MCP reliability; one could monitor issue trackers to test whether discovery faults cause outsized user impact.","Because the sample is Python-only and collected soon after MCP's release, the taxonomy may underrepresent faults that emerge with maturity, such as security issues and version skew across SDKs; extending the same coding to TypeScript SDKs and later time windows is a natural check."],"forward_implications":["MCP server testing efforts should target configuration and integration seams—dependencies, tool registration, host connection settings—rather than only tool logic.","Issue triage tools for MCP repositories can use the five categories as a checklist to route and prioritize bug reports.","MCP-specific fault patterns being distinct from generic bugs means automated debugging and repair should be trained on MCP-aware data, not generic software faults.","The observed short fix times and lower fixer experience for MCP faults suggest a small pool of domain specialists handles them; improving documentation and defaults could reduce that bottleneck."],"fun_headline_variants":["Five fault categories map where MCP servers break in practice","Configuration seams are the leading MCP server fault source","MCP faults attract more chatter yet get fixed faster, study finds","First large-scale taxonomy of MCP server faults reveals five types","Tool response handling and docs top MCP fault survey list"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on the assumption that the 407 issues selected by the LLM-assisted classification and clustering pipeline faithfully represent real MCP faults; the pipeline's accuracy was checked on only 40 hand-labeled issues, and clustering-relatedness was judged on 20 issues per cluster, so sampling or labeling errors could reshape the taxonomy.","fun_headline_variants_meta":{"raw":{"variants":["Five fault categories map where MCP servers break in practice","Configuration seams are the leading MCP server fault source","MCP faults attract more chatter yet get fixed faster, study finds","First large-scale taxonomy of MCP server faults reveals five types","Tool response handling and docs top MCP fault survey list"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3242,"prompt_tokens":758,"completion_tokens":2484,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":2401}},"tokens_in":502,"tokens_out":2484,"duration_ms":19554,"temperature":1.0,"reasoning_tokens":2401,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T05:51:35.661044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Label a fresh random sample of 200 closed bug issues from MCP server projects written in non-Python SDKs using the paper's codebook. If a substantial fraction (say more than 15-20%) of MCP-related bugs fall outside the five categories, or if the category distribution shifts dramatically, the taxonomy's completeness and generalizability claims fail.","supporting_citations":[],"review_version":1}