Pith. sign in

REVIEW 4 major objections 7 minor 7 cited by

RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieving relevant tool descriptions before prompting an LLM cuts prompt tokens by over half and more than triples MCP tool-selection accuracy.

desk verdict The idea is plausible and the stress test is a useful evaluation pattern, but the quantitative claims as reported are internally inconsistent and the scalability conclusion is contradicted by the paper's own stress test. read the letter →

arxiv 2505.03275 v1 pith:WP3SBP3E submitted 2025-05-06 cs.AI cs.SE

Tiantian Gan , Qiyao Sun This is my paper · ORCID
classification cs.AIcs.SE
keywords retrieval-augmentedgenerationModelContextProtocoltoolselectionpromptbloatfunctioncallingLLMagentssemanticretrievalMCPstresstest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes RAG-MCP, a retrieval-augmented pipeline for LLM tool use under the Model Context Protocol. Instead of placing every available MCP tool description into the prompt, the system first semantically retrieves the most relevant tool from an external index and passes only that schema to the LLM. The paper reports that this reduces prompt tokens by more than half and raises tool-selection accuracy from 13.62% to 43.13% on the MCPBench web-search subset. A stress test varying the candidate pool from 1 to 11,100 MCPs shows the approach holds up in small and moderate pools but degrades beyond roughly 100 candidates, which the authors acknowledge and flag for future hierarchical retrieval.

What carries the argument

The load-bearing object is the semantic vector index over MCP server descriptions plus the top-k retrieval step, with k equal to one in deployment. The retriever embeds both the query and the tool schemas, ranks candidates by semantic similarity, and injects only the selected schema into the LLM prompt. This turns tool selection from a needle-in-a-haystack problem over thousands of distractors into a focused one-choice decision, which is what cuts token usage and improves accuracy. An optional validation step generates a synthetic example query to sanity-check compatibility before invocation.

What would settle it

Run the MCP stress test with N larger than a few hundred and count how often the ground-truth MCP appears in the top-1 retrieval result. If top-1 recall stays high, say above 80%, at N greater than 500, the paper's stress-test conclusion is wrong; if it falls sharply, the scalability claim fails exactly as the stress test shows. A second check is to hold N fixed and shuffle or reword MCP descriptions, which would test whether the accuracy gain comes from retrieval itself or merely from shortening the prompt.

Watch

Extended reading notes

Core claim

The central claim is that offloading tool discovery to a semantic retriever makes LLM tool use scale better than naive all-tools-in-context prompting. Concretely, RAG-MCP maintains a vector index of MCP schemas, encodes the user's task, retrieves the top candidate, and only then gives the LLM that single tool definition. In experiments on MCPBench web-search tasks the paper reports 43.13% selection accuracy versus 13.62% for blank conditioning and 18.20% for keyword matching, while average prompt tokens fall from 2,133.84 to 1,084.00. The paper also constructs an MCP stress test with up to 11,100 servers and shows that retrieval precision is high at small registry sizes but fails beyond roughly position 100, so the scalable claim is explicitly bounded by retriever quality at large registry sizes.

Load-bearing premise

The whole benefit depends on the retriever returning the true MCP in its top pick; if the retriever misses it, the LLM can never select the right tool, and the paper's own stress test shows retrieval precision collapses beyond roughly 100 candidates.

Editorial extensions

If this is right

  • New tools can be added to the external index without retraining the LLM, so the deployment can stay current without model updates.
  • On the web-search benchmark, operators should expect prompt tokens near 1,084 versus 2,134 for all-tools prompting, and selection accuracy near 43% versus 13.6% for the blank baseline.
  • Because the stress test shows retrieval precision falling sharply past roughly 100 candidate MCPs, RAG-MCP as designed is not yet a full solution for very large registries; the paper itself flags hierarchical or adaptive retrieval as future work.
  • Selectively activating only the retrieved MCP avoids instantiating every registered server at startup, lowering the resource cost of supporting large tool pools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same retrieve-then-prompt pattern should transfer to other tool ecosystems, including plugin APIs and home-grown function libraries, because the mechanism only assumes searchable text descriptions of tools.
  • A natural next experiment is to vary semantic overlap among MCP descriptions directly; if retrieval failures track overlap rather than raw N, then schema quality and embedding distinctness, not registry size alone, determine the scaling limit.
  • The completion-token increase reported for RAG-MCP (78.14 vs 23.60) suggests the single retrieved schema invites more extended reasoning; an end-to-end task-success metric would clarify whether that extra generation actually improves final answers.
  • The stress-test data could be refit as a retrieval-recall-versus-log-N curve; the apparent transition near N≈100 would make a concrete target for a hierarchical index to beat.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes RAG-MCP, a retrieval-augmented framework for large language model tool selection in the Model Context Protocol setting. MCP server descriptions are indexed in a vector store; at query time a retriever selects the most relevant candidate and only that tool's schema is injected into the LLM prompt, reducing prompt bloat and simplifying the model's selection decision. The authors evaluate on the web-search subset of MCPBench against a no-filter 'Blank' baseline and a keyword-based 'Actual Match' baseline, and they also report an 'MCP stress test' in which the number of candidate MCPs is varied. The paper claims a prompt-token reduction of over 50%, an accuracy of 43.13% for RAG-MCP versus 13.62% for Blank, and scalability to very large tool registries.

Significance. The problem addressed here is real and timely: LLM agents increasingly face large, heterogeneous tool registries, and naive prompting with all tool descriptions is both token-expensive and error-prone. The central idea, decoupling tool discovery from tool invocation via retrieval, is plausible and potentially useful, and the choice to evaluate on a public benchmark (MCPBench) plus a purpose-built stress test is commendable. The paper also emphasizes practical advantages such as adding tools to an index without retraining. However, as reported, the evidence does not support the central quantitative claims. The accuracy figures in Table 1 are arithmetically incompatible with the stated trial count; the stress test as described is internally impossible at its stated maximum N; the paper's own stress-test analysis shows retrieval collapse in exactly the regime where the abstract claims scalability; and no code or data release is mentioned. These are load-bearing problems for the paper's headline claims, so the manuscript cannot be accepted in its current form.

major comments (4)
  1. [Section 4.2, Table 1] The stated protocol of 20 independent trials per baseline makes the reported accuracies 43.13%, 18.20%, and 13.62% impossible: with 20 binary trials, accuracy must be a multiple of 5%, corresponding to 8.626, 3.64, and 2.724 correct trials, respectively. The sentence 'we deem a baseline successful if it produces more than 10 correct answers out of those 20' is also inconsistent with a 43.13% result. The authors must report raw counts, define the denominator precisely, and reconcile the success criterion with the accuracy metric; as written, the headline accuracy comparison is unsupported.
  2. [Section 3.2 and Section 5, Figure 3] RAG-MCP injects only the top-1 retrieved MCP into the prompt, so overall tool-selection accuracy is upper-bounded by top-1 retrieval recall. Figure 3 and the accompanying text state that beyond position ~100 'purple dominates, signifying that retrieval precision diminishes,' which directly contradicts the abstract's claim that RAG-MCP 'enables scalable' tool integration. The MCPBench evaluation in Table 1 does not report the registry size N used in the benchmark, so there is no evidence that the method works outside the small-N regime where the stress test already shows retrieval success. The authors need to report N for the benchmark and either demonstrate acceptable retrieval recall at large N or revise the scalability claim.
  3. [Section 4.1] The stress test is described as varying N from 1 to 11100 while drawing N−1 distractor MCPs from 'our full registry of over 4,400 publicly listed servers.' For N greater than roughly 4,401, there are not enough distinct distractors to populate the pool as described, so the stress-test construction is internally inconsistent at large N. The authors should clarify whether distractors are sampled with replacement, whether the maximum N is limited, or whether 'positions' are defined differently; without this, the stress-test results cannot be interpreted at scale.
  4. [Abstract and Table 1] The abstract claims that RAG-MCP cuts prompt tokens 'by over 50%,' but Table 1 reports average prompt tokens of 2133.84 for Blank and 1084.00 for RAG-MCP, which is a 49.2% reduction, not 'over half.' The conclusion repeats the 'over half' formulation. The quantitative claim should be corrected to match the reported data, or the experimental protocol should be changed so that the reported reduction actually exceeds 50%.
minor comments (7)
  1. [Section 4.2, Metrics] The setup paragraph says answer correctness is judged using Deepseek-v3, while the Metrics paragraph says a Llama-based verifier is used. These two statements should be reconciled or the complementary roles of the two models should be explained.
  2. [Section 4.2, Metrics] Accuracy is defined as the percentage of trials in which the model selected the ground-truth MCP, but each trial allows up to 10 rounds of interaction with MCP servers. It is not stated whether a correct selection in any round counts, or only the first or final selection. This ambiguity affects interpretation of all accuracy numbers.
  3. [Table 1] The table header uses 'MCP-RAG' although the method is named 'RAG-MCP' throughout the paper; the terminology should be standardized.
  4. [Section 3.1] The Needle-in-a-Haystack analogy cites references [6] and [10], which are the RAG paper and an OpenAI function-calling page, not a source for the NIAH test; the citation should be corrected.
  5. [Figure 3] Figure 3 is described only verbally in the text; the printed figure should have explicit axis labels, a legend for the yellow/purple color coding, and a clear statement of the 26 N-intervals used.
  6. [Throughout] No standard errors or confidence intervals are reported for the accuracy values, which is problematic given the small stated number of trials; at minimum the raw counts should be given.
  7. [Throughout] Proofreading is needed: 'Comparation' in the Figure 1 caption, 'an suboptimal' in Section 1.1, and the mixed use of 'MCP-RAG' and 'RAG-MCP' should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the framework is evaluated against an external benchmark and none of its predictions are constructed from its own outputs.

full rationale

RAG-MCP is a retrieval-augmented tool-selection method whose central claim is evaluated on the external MCPBench web-search subset against two baselines. The reported accuracy figures are measurement outcomes, not quantities defined so as to reproduce the method's inputs; the retriever is not fitted to the benchmark, and no parameter is renamed as a prediction. The paper contains no load-bearing self-citation: the cited prior work (Gorilla, Toolformer, ReAct, RAG, MCPBench report) is external, and the authors' own contributions are not justified by an appeal to their own unpublished results. The stress-test analysis in Section 4.1 is an empirical observation about retrieval degradation with registry size, not a definitional identity that forces the headline result. Even if the trial-count/accuracy inconsistency and evaluator description mismatch noted by the skeptic indicate correctness or reproducibility risks, those are internal-consistency problems rather than circularity: the derivation chain does not reduce the claimed prediction to its own inputs by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or formal entities are postulated. The method relies on several domain assumptions about retrieval quality, benchmark labeling, and judge reliability, all of which are plausible but not independently validated.

free parameters (1)
  • top_k (number of MCP descriptions injected into prompt) = 1
    Chosen by hand; the paper states 'Only the single best MCP description' is passed to the model. No sensitivity analysis is given, and this choice directly determines the reported prompt-size reduction.
assumptions (4)
  • domain assumption Semantic vector retrieval over MCP descriptions reliably ranks the ground-truth tool above distractors for the benchmark queries.
    The method depends on the embedding space capturing query-tool relevance; it is invoked throughout Section 3.2 and is assessed only via Figure 3, which shows strong degradation at large N.
  • domain assumption The MCPBench web-search subset provides correct ground-truth tool labels for each query.
    The heldout testbed is taken from reference [8]; its labeling accuracy is not independently verified in this paper.
  • domain assumption The automated judge (Deepseek-v3 or the 'Llama as Judge' verifier) scores answer correctness reliably.
    Section 4.2 first states Deepseek-v3 is the evaluator, then says a Llama-based verifier judges answers. The judge's agreement with humans is not measured.
  • domain assumption Exactly one MCP server in the candidate pool is capable of satisfying each web-search task.
    Stress test design (Section 4.1) assumes one ground-truth MCP and N-1 distractors; in a registry of 4,400 servers with many overlapping web-search tools, this exclusivity is not evident.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/WP3SBP3E

@misc{pith2026250503275,
  author       = {Pith},
  title        = {Pith review of: RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WP3SBP3E}},
  note         = {Machine review of arXiv:2505.03275}
}
read the original abstract

Large language models (LLMs) struggle to effectively utilize a growing number of external tools, such as those defined by the Model Context Protocol (MCP)\cite{IntroducingMCP}, due to prompt bloat and selection complexity. We introduce RAG-MCP, a Retrieval-Augmented Generation framework that overcomes this challenge by offloading tool discovery. RAG-MCP uses semantic retrieval to identify the most relevant MCP(s) for a given query from an external index before engaging the LLM. Only the selected tool descriptions are passed to the model, drastically reducing prompt size and simplifying decision-making. Experiments, including an MCP stress test, demonstrate RAG-MCP significantly cuts prompt tokens (e.g., by over 50%) and more than triples tool selection accuracy (43.13% vs 13.62% baseline) on benchmark tasks. RAG-MCP enables scalable and accurate tool integration for LLMs.

Figures

Figures reproduced from arXiv: 2505.03275 by the authors.

Figure 1
Figure 1. Comparation between MCP and RAG-MCP during inference To tackle these challenges, we propose RAG-MCP, a solution that mar￾ries Retrieval-Augmented Generation (RAG) with the Model Context Protocol framework. The key idea of RAG-MCP is to avoid presenting all tools to the lan￾guage model at once, and instead dynamically retrieve a relevant subset of tools based on the user’s query. In our approach, the numerous availab… view at source ↗
Figure 2
Figure 2. RAG-MCP pipeline: (1) encode user query with Qwen-max, (2) re-trieve & validate top-k MCPs, and (3) invoke chosen MCP By decoupling tool discovery from generation, RAG-MCP ensures that LLMs can scale to hundreds or thousands of MCPs without suffering prompt bloat or decision fatigue, much as RAG systems avoid overwhelming an LLM with entire corpora by retrieving only relevant passages. 3.4 Discussion Our methodology… view at source ↗
Figure 3
Figure 3. This figure illustrates per-trial success across MCP positions from 1 to 11100, where yellow denotes successful selection and purple denotes failure [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Constrained decoding derived from OpenAPI specifications eliminates hallucinated web API calls in LLM-generated code and substantially improves correctness across 24 models, while retrieval-augmented generation yields...

  2. MCP-Zero: Active Tool Discovery for Autonomous LLM Agents

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An LLM agent framework where the model actively emits structured server/tool requests, retrieved through hierarchical semantic routing, reducing context overhead while maintaining tool-selection accuracy.

  3. On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models

    cs.CR 2026-08 conditional novelty 5.0 of 10

    A PRISMA-based survey of 85 papers shows agentic LLM security research is attack-heavy and perception-focused, leaving action-layer and code-execution risks understudied.

  4. Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale

    cs.AI 2026-08 conditional novelty 5.0 of 10

    In Wix's deployed Helpmate assistant, a deterministic executability gate removed 59.4% of semantically matched skill candidates and, in a counterfactual replay, prevented selection of production-blocked skills in 7.8%...

  5. Adapting Embedding Models for Agent Capability Retrieval

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Fine-tuning three off-the-shelf retrieval models on AgentSelect improved query-to-agent ranking on two unseen marketplace catalogs, MuleRun and ClawHub, across all three model families.

  6. Scalable LLM Agent Tool Access in the Cloud

    cs.DC 2026-07 conditional novelty 5.0 of 10

    A cloud-scale MCP gateway with hybrid dense-sparse retrieval lets LLM agents work with 3,000+ tools at 98% Top-15 recall, cutting tool-selection time 8.9× and token use 23.8×.

  7. Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution

    cs.AI 2025-05 reject novelty 5.0 of 10

    Alita achieves 75.15% pass@1 on GAIA validation by dynamically generating and reusing MCP-based tools, but the evaluation confounds model choice with the self-evolution mechanism.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages · cited by 7 Pith papers

  1. [1]

    anthropic.com/news/model-context-protocol

    Anthropic: Introducing the model context protocol (2024), https://www. anthropic.com/news/model-context-protocol

    work page 2024

  2. [2]

    nvidia.com/blog/what-is-retrieval-augmented-generation/

    Blog, N.: What is retrieval-augmented generation aka rag (2025),https://blogs. nvidia.com/blog/what-is-retrieval-augmented-generation/

    work page 2025

  3. [3]

    arXiv preprint arXiv:2412.01130 (2024)

    Chen, Y.C., Hsu, P.C., Hsu, C.J., Shiu, D.s.: Enhancing function-calling capa- bilities in llms: Strategies for prompt formats, data integration, and multilingual translation. arXiv preprint arXiv:2412.01130 (2024)

    arXiv 2024

  4. [4]

    arXiv preprint arXiv:2312.109972 (2023) 12 Gan and Sun

    Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H., Wang, H.: Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.109972 (2023) 12 Gan and Sun

    arXiv 2023

  5. [5]

    gkamradt: The needle in a haystack test (2024),https://github.com/gkamradt/ LLMTest_NeedleInAHaystack

    work page 2024

  6. [6]

    In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küt- tler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval- augmented generation for knowledge-intensive nlp tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 945...

    work page 2020

  7. [7]

    arXiv preprint arXiv:2412.19437 (2024)

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    arXiv 2024

  8. [8]

    Luo, Z., Shi, X., Lin, X., Gao, J.: Evaluation report on mcp servers (2025),https: //arxiv.org/abs/2504.11094

    arXiv 2025

Show all 15 references
  1. [9]

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., Schulman, J.: Webgpt: Browser-assisted question-answering with human feedback (...

    2022

  2. [10]

    OpenAI: Openai function calling, https://platform.openai.com/docs/guides/ function-calling

  3. [11]

    OpenAI: Chatgpt plugins (2023),https://openai.com/index/chatgpt-plugins

    2023

  4. [12]

    Advances in Neural Information Processing Systems 37, 126544–126565 (2024)

    Patil, S.G., Zhang, T., Wang, X., Gonzalez, J.E.: Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems 37, 126544–126565 (2024)

    2024

  5. [13]

    In: Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., Levine, S

    Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., Scialom, T.: Toolformer: Language mod- els can teach themselves to use tools. In: Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances...

    2023

  6. [14]

    ShipAny: Mcp servers (2025),https://mcp.so/

    2025

  7. [15]

    Evaluation Report on MCP Servers

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: Re- act: Synergizing reasoning and acting in language models. International Confer- ence on Learning Representations (ICLR) (2023),https://par.nsf.gov/biblio/ 10451467 7 Acknowledgements Wegratefullyackno...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.