{"id":"05a14407-cabd-4896-b8f8-03394d209731","arxiv_id":"2505.03275","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"RAG-MCP retrieves the most relevant Model Context Protocol server description for each query and feeds only that schema to the LLM, reporting fewer prompt tokens and higher selection accuracy on a small benchmark.","lead":"This paper proposes RAG-MCP, a system that retrieves only the most relevant tool descriptions from a searchable index before asking a language model to choose a tool. A generalist would read it because it targets a practical bottleneck for AI agents that must handle dozens or thousands of plug-in tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark numbers are internally inconsistent: 20 trials cannot produce 43.13%; the paper's own stress test shows retrieval collapse beyond ~100 tools, undermining scalability.","rationale":"The reader correctly identified the retriever top-1 ceiling as a key risk; Figure 3 itself shows success collapsing beyond roughly position 100, directly contradicting the abstract's 'scalable' wording. I think that is a genuine concern, and the lack of a reported registry size N in Table 1 makes it impossible to tell whether the benchmark runs in the regime where RAG-MCP works or the regime where it fails. I would rank the internal counting inconsistency as equally load-bearing: 20 trials cannot yield the reported accuracy percentages, so the quantitative claims are not well-defined as stated. That is a correctness risk, not a stylistic quibble, because it blocks verification of every headline number. The evaluator contradiction (Deepseek-v3 vs. Llama) and the abstract-vs-table token reduction (49.2% rather than 'over 50%') further reduce confidence. These are not disagreements with consensus; they are failures of internal consistency. The proposed system concept is plausible and modestly useful, but the evidence as written does not support the central claims. Therefore REJECT remains the appropriate verdict, and my stress-test pass does not change the reader's conclusion.","tokens_in":7743,"tokens_out":3150,"duration_ms":30146,"concrete_test":"Release the per-trial raw outcome table for all baselines (20 trials per condition), recompute Table 1 accuracy as the number of correct trials divided by 20, and report the exact registry size N used for the MCPBench runs. If 43.13% is not a multiple of 5% for a 20-trial count, or if the MCPBench N exceeds ~100 and top-1 retrieval recall is not high, then the discrepancy is resolved; otherwise the central accuracy and scalability claims fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claim (abstract: 'over 50%' token cut and 43.13% vs 13.62% accuracy) is not supported by internally consistent evidence. Section 4.2 states each baseline is run for 20 independent trials; with 20 trials, accuracy can only take values in 5% increments. Table 1 reports 43.13%, 18.20%, and 13.62%, which would require 8.626, 3.64, and 2.724 correct trials respectively. Thus either the trial count, the denominator, or the numbers are wrong. The evaluator is also described twice: 'Deepseek-v3 as our evaluator' in Setup but 'Llama as Judge' in Metrics; and the abstract's 'over 50%' reduction is 49.2% from Table 1. Separately, even if those numbers were fixed, Section 4.1/Figure 3 (the authors' own stress test) shows success 'beyond position ~100 purple dominates, signifying retrieval precision diminishes'; since RAG-MCP injects only the top-1 retrieved MCP, its accuracy is capped by top-1 retrieval recall. The main benchmark's registry size N is not reported, so the central 'scalable' claim rests on an unshown, likely small-N regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAG-MCP, a retrieval-augmented framework for large language model tool selection in the Model Context Protocol setting. MCP server descriptions are indexed in a vector store; at query time a retriever selects the most relevant candidate and only that tool's schema is injected into the LLM prompt, reducing prompt bloat and simplifying the model's selection decision. The authors evaluate on the web-search subset of MCPBench against a no-filter 'Blank' baseline and a keyword-based 'Actual Match' baseline, and they also report an 'MCP stress test' in which the number of candidate MCPs is varied. The paper claims a prompt-token reduction of over 50%, an accuracy of 43.13% for RAG-MCP versus 13.62% for Blank, and scalability to very large tool registries.","tokens_in":7880,"tokens_out":6263,"duration_ms":59137,"significance":"The problem addressed here is real and timely: LLM agents increasingly face large, heterogeneous tool registries, and naive prompting with all tool descriptions is both token-expensive and error-prone. The central idea, decoupling tool discovery from tool invocation via retrieval, is plausible and potentially useful, and the choice to evaluate on a public benchmark (MCPBench) plus a purpose-built stress test is commendable. The paper also emphasizes practical advantages such as adding tools to an index without retraining. However, as reported, the evidence does not support the central quantitative claims. The accuracy figures in Table 1 are arithmetically incompatible with the stated trial count; the stress test as described is internally impossible at its stated maximum N; the paper's own stress-test analysis shows retrieval collapse in exactly the regime where the abstract claims scalability; and no code or data release is mentioned. These are load-bearing problems for the paper's headline claims, so the manuscript cannot be accepted in its current form.","major_comments":[{"comment":"The stated protocol of 20 independent trials per baseline makes the reported accuracies 43.13%, 18.20%, and 13.62% impossible: with 20 binary trials, accuracy must be a multiple of 5%, corresponding to 8.626, 3.64, and 2.724 correct trials, respectively. The sentence 'we deem a baseline successful if it produces more than 10 correct answers out of those 20' is also inconsistent with a 43.13% result. The authors must report raw counts, define the denominator precisely, and reconcile the success criterion with the accuracy metric; as written, the headline accuracy comparison is unsupported.","section":"Section 4.2, Table 1"},{"comment":"RAG-MCP injects only the top-1 retrieved MCP into the prompt, so overall tool-selection accuracy is upper-bounded by top-1 retrieval recall. Figure 3 and the accompanying text state that beyond position ~100 'purple dominates, signifying that retrieval precision diminishes,' which directly contradicts the abstract's claim that RAG-MCP 'enables scalable' tool integration. The MCPBench evaluation in Table 1 does not report the registry size N used in the benchmark, so there is no evidence that the method works outside the small-N regime where the stress test already shows retrieval success. The authors need to report N for the benchmark and either demonstrate acceptable retrieval recall at large N or revise the scalability claim.","section":"Section 3.2 and Section 5, Figure 3"},{"comment":"The stress test is described as varying N from 1 to 11100 while drawing N−1 distractor MCPs from 'our full registry of over 4,400 publicly listed servers.' For N greater than roughly 4,401, there are not enough distinct distractors to populate the pool as described, so the stress-test construction is internally inconsistent at large N. The authors should clarify whether distractors are sampled with replacement, whether the maximum N is limited, or whether 'positions' are defined differently; without this, the stress-test results cannot be interpreted at scale.","section":"Section 4.1"},{"comment":"The abstract claims that RAG-MCP cuts prompt tokens 'by over 50%,' but Table 1 reports average prompt tokens of 2133.84 for Blank and 1084.00 for RAG-MCP, which is a 49.2% reduction, not 'over half.' The conclusion repeats the 'over half' formulation. The quantitative claim should be corrected to match the reported data, or the experimental protocol should be changed so that the reported reduction actually exceeds 50%.","section":"Abstract and Table 1"}],"minor_comments":[{"comment":"The setup paragraph says answer correctness is judged using Deepseek-v3, while the Metrics paragraph says a Llama-based verifier is used. These two statements should be reconciled or the complementary roles of the two models should be explained.","section":"Section 4.2, Metrics"},{"comment":"Accuracy is defined as the percentage of trials in which the model selected the ground-truth MCP, but each trial allows up to 10 rounds of interaction with MCP servers. It is not stated whether a correct selection in any round counts, or only the first or final selection. This ambiguity affects interpretation of all accuracy numbers.","section":"Section 4.2, Metrics"},{"comment":"The table header uses 'MCP-RAG' although the method is named 'RAG-MCP' throughout the paper; the terminology should be standardized.","section":"Table 1"},{"comment":"The Needle-in-a-Haystack analogy cites references [6] and [10], which are the RAG paper and an OpenAI function-calling page, not a source for the NIAH test; the citation should be corrected.","section":"Section 3.1"},{"comment":"Figure 3 is described only verbally in the text; the printed figure should have explicit axis labels, a legend for the yellow/purple color coding, and a clear statement of the 26 N-intervals used.","section":"Figure 3"},{"comment":"No standard errors or confidence intervals are reported for the accuracy values, which is problematic given the small stated number of trials; at minimum the raw counts should be given.","section":"Throughout"},{"comment":"Proofreading is needed: 'Comparation' in the Figure 1 caption, 'an suboptimal' in Section 1.1, and the mixed use of 'MCP-RAG' and 'RAG-MCP' should be fixed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper's dependence on the MCPBench dataset and code is acknowledged, but the mismatch between the reported accuracy granularity and the stated number of trials suggests the experimental records are not internally consistent. In addition, the authors' own stress test undercuts the abstract's scalability claim, and the benchmark registry size is not reported. I would not invite resubmission unless the experiments are redone with a clear protocol, the arithmetic is corrected, and the claims are restricted to the regime the data actually support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper applies a straightforward RAG idea to MCP tool selection, and it does that clearly, but the evidence section does not support the headline numbers. The problem it addresses is real — prompt bloat in tool-rich agents is an active pain point — and the NIAH-inspired stress test is a genuinely sensible way to probe degradation as the tool registry grows. Credit where due: the framing is clean, the pipeline is simple, and the authors correctly cite Gorilla as prior art for retrieving API documentation. Extending that to MCP servers and adding a stress test is a legitimate incremental step, not a breakthrough.\n\nThe soft spots are serious. Twenty independent trials cannot yield accuracies of 43.13%, 18.20%, and 13.62% — those require non-integer counts of correct trials. Either the trial count, the denominator, or the numbers are wrong. The evaluator is described as Deepseek-v3 in one section and as a Llama-based judge in another. The abstract says token reduction is “over 50%” while Table 1 shows 49.2%. These are not cosmetic; they mean every quantitative claim in the paper is suspect. On top of that, the authors' own stress test (Figure 3) shows retrieval precision collapsing beyond roughly 100 MCPs, yet the main benchmark does not report the registry size N. Since RAG-MCP injects only the top-1 retrieved MCP, its accuracy is capped by top-1 retrieval recall, so the “scalable” part of the central claim is unsupported. No code, data, or prompts are released, so independent verification is impossible.\n\nThe citation pattern is thin but not misleading; the related work covers the main anchors (Gorilla, Toolformer, ReAct, RAG). The conceptual contribution is modest — this is RAG applied to a new protocol — but the stress-test design could be useful to the agents community if the authors repair the evaluation and release artifacts.\n\nBottom line: I would not trust any of the reported numbers in the current draft, but I would not desk-reject it either. The topic is timely, the idea is reasonable, and a serious referee could help the authors turn this into something citable. My recommendation: send it to review, but flag the internal inconsistencies prominently and require corrected statistics, a reported N, and artifact release before any acceptance.\n\nWho is this for? Someone working on tool selection or MCP integration might read it for the stress-test idea, but they should not cite the accuracy numbers until the authors fix the reporting.","headline":"The idea is plausible and the stress test is a useful evaluation pattern, but the quantitative claims as reported are internally inconsistent and the scalability conclusion is contradicted by the paper's own stress test.","tokens_in":8527,"tokens_out":1952,"would_cite":false,"duration_ms":19982,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieving relevant tool descriptions before prompting an LLM cuts prompt tokens by over half and more than triples MCP tool-selection accuracy.","keywords":["retrieval-augmented generation","Model Context Protocol","tool selection","prompt bloat","function calling","LLM agents","semantic retrieval","MCP stress test"],"falsifier":"Run the MCP stress test with N larger than a few hundred and count how often the ground-truth MCP appears in the top-1 retrieval result. If top-1 recall stays high, say above 80%, at N greater than 500, the paper's stress-test conclusion is wrong; if it falls sharply, the scalability claim fails exactly as the stress test shows. A second check is to hold N fixed and shuffle or reword MCP descriptions, which would test whether the accuracy gain comes from retrieval itself or merely from shortening the prompt.","tokens_in":7411,"feed_emoji":"🔧","tokens_out":5486,"duration_ms":47398,"temperature":0.7,"pith_summary":"This paper proposes RAG-MCP, a retrieval-augmented pipeline for LLM tool use under the Model Context Protocol. Instead of placing every available MCP tool description into the prompt, the system first semantically retrieves the most relevant tool from an external index and passes only that schema to the LLM. The paper reports that this reduces prompt tokens by more than half and raises tool-selection accuracy from 13.62% to 43.13% on the MCPBench web-search subset. A stress test varying the candidate pool from 1 to 11,100 MCPs shows the approach holds up in small and moderate pools but degrades beyond roughly 100 candidates, which the authors acknowledge and flag for future hierarchical retrieval.","feed_headline":"Retrieval-first tool selection triples accuracy, halves tokens","feed_subtitle":"Retrieving only the relevant MCP schema cuts prompt tokens by over half and lifts tool choice from 13.6% to 43.1%.","key_machinery":"The load-bearing object is the semantic vector index over MCP server descriptions plus the top-k retrieval step, with k equal to one in deployment. The retriever embeds both the query and the tool schemas, ranks candidates by semantic similarity, and injects only the selected schema into the LLM prompt. This turns tool selection from a needle-in-a-haystack problem over thousands of distractors into a focused one-choice decision, which is what cuts token usage and improves accuracy. An optional validation step generates a synthetic example query to sanity-check compatibility before invocation.","core_discovery":"The central claim is that offloading tool discovery to a semantic retriever makes LLM tool use scale better than naive all-tools-in-context prompting. Concretely, RAG-MCP maintains a vector index of MCP schemas, encodes the user's task, retrieves the top candidate, and only then gives the LLM that single tool definition. In experiments on MCPBench web-search tasks the paper reports 43.13% selection accuracy versus 13.62% for blank conditioning and 18.20% for keyword matching, while average prompt tokens fall from 2,133.84 to 1,084.00. The paper also constructs an MCP stress test with up to 11,100 servers and shows that retrieval precision is high at small registry sizes but fails beyond roughly position 100, so the scalable claim is explicitly bounded by retriever quality at large registry sizes.","pith_inferences":["The same retrieve-then-prompt pattern should transfer to other tool ecosystems, including plugin APIs and home-grown function libraries, because the mechanism only assumes searchable text descriptions of tools.","A natural next experiment is to vary semantic overlap among MCP descriptions directly; if retrieval failures track overlap rather than raw N, then schema quality and embedding distinctness, not registry size alone, determine the scaling limit.","The completion-token increase reported for RAG-MCP (78.14 vs 23.60) suggests the single retrieved schema invites more extended reasoning; an end-to-end task-success metric would clarify whether that extra generation actually improves final answers.","The stress-test data could be refit as a retrieval-recall-versus-log-N curve; the apparent transition near N≈100 would make a concrete target for a hierarchical index to beat."],"forward_implications":["New tools can be added to the external index without retraining the LLM, so the deployment can stay current without model updates.","On the web-search benchmark, operators should expect prompt tokens near 1,084 versus 2,134 for all-tools prompting, and selection accuracy near 43% versus 13.6% for the blank baseline.","Because the stress test shows retrieval precision falling sharply past roughly 100 candidate MCPs, RAG-MCP as designed is not yet a full solution for very large registries; the paper itself flags hierarchical or adaptive retrieval as future work.","Selectively activating only the retrieved MCP avoids instantiating every registered server at startup, lowering the resource cost of supporting large tool pools."],"supporting_citations":[{"why":"Defines the Model Context Protocol, the standardized tool format that RAG-MCP retrieves and invokes.","marker":"[1]"},{"why":"Supplies the needle-in-a-haystack evaluation idea that the paper adapts into its MCP stress test.","marker":"[5]"},{"why":"Provides the retrieval-augmented generation formulation and dense vector index concept that the framework builds on.","marker":"[6]"},{"why":"Serves as the automated evaluator used to judge whether final answers are correct in the experiments.","marker":"[7]"},{"why":"Supplies the MCPBench web-search subset used as the held-out testbed for the main accuracy comparison.","marker":"[8]"},{"why":"Shows that augmenting an LLM with retrieved API documentation improves tool-call accuracy, motivating retrieval over all available tools.","marker":"[12]"},{"why":"Provides the public MCP registry of over 4,400 servers from which stress-test distractors are drawn.","marker":"[14]"}],"fun_headline_variants":["Retrieval-first tool selection triples accuracy, halves tokens","RAG-MCP: 3x tool accuracy with half the prompt tokens","Retrieve the right tool: accuracy triples, prompt halves","LLM tool selection: retrieve, don't overload the prompt","Prompt bloat? Retrieve your tools and triple accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benefit depends on the retriever returning the true MCP in its top pick; if the retriever misses it, the LLM can never select the right tool, and the paper's own stress test shows retrieval precision collapses beyond roughly 100 candidates.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-first tool selection triples accuracy, halves tokens","RAG-MCP: 3x tool accuracy with half the prompt tokens","Retrieve the right tool: accuracy triples, prompt halves","LLM tool selection: retrieve, don't overload the prompt","Prompt bloat? Retrieve your tools and triple accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2090,"prompt_tokens":902,"completion_tokens":1188,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1101}},"tokens_in":518,"tokens_out":1188,"duration_ms":11059,"temperature":1.0,"reasoning_tokens":1101,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:55:30.848656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MCP stress test with N larger than a few hundred and count how often the ground-truth MCP appears in the top-1 retrieval result. If top-1 recall stays high, say above 80%, at N greater than 500, the paper's stress-test conclusion is wrong; if it falls sharply, the scalability claim fails exactly as the stress test shows. A second check is to hold N fixed and shuffle or reword MCP descriptions, which would test whether the accuracy gain comes from retrieval itself or merely from shortening the prompt.","supporting_citations":[{"cited_title":"anthropic.com/news/model-context-protocol","cited_arxiv_id":null,"evidence_quote":"Defines the Model Context Protocol, the standardized tool format that RAG-MCP retrieves and invokes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the needle-in-a-haystack evaluation idea that the paper adapts into its MCP stress test."},{"cited_title":"In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H","cited_arxiv_id":null,"evidence_quote":"Provides the retrieval-augmented generation formulation and dense vector index concept that the framework builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the public MCP registry of over 4,400 servers from which stress-test distractors are drawn."}],"review_version":1}