{"id":"a16039d0-5098-4395-ae79-ffdb416404cc","arxiv_id":"2607.15593","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A cloud-scale MCP gateway with hybrid dense-sparse retrieval lets LLM agents work with 3,000+ tools at 98% Top-15 recall, cutting tool-selection time 8.9× and token use 23.8×.","lead":"This paper describes a production gateway that sits between AI agents and the tools they call: it converts legacy web APIs into the new MCP format, retrieves the most relevant tools instead of stuffing all of them into the model's context, and keeps multi-replica sessions consistent. Headline results from an eight-month Alibaba Cloud deployment: 98% Top-15 recall at 3,616 tools, 8.9× faster tool selection, 23.8× fewer tokens.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels for Recall@15 may come from agents already using the gateway's Tool Search, making the 98.2% recall self-confirmatory; the paper's own §7 states production agents rely on the recommendation during the traced period.","rationale":"The reader identified the same load-bearing concern: the ground-truth labels in the production traces may not be independent of the retrieval pipeline being evaluated. The paper's own §7 provides direct support for this worry: in the production traces, agents rely on the gateway's tool recommendation for tool discovery, so the 'tool actually invoked' is very likely a tool the gateway recommended. This creates a circularity that would inflate both the absolute recall (98.2%) and the relative advantage of hybrid over dense-only retrieval. The reader also noted a secondary fragility about transferability to generic tool descriptions, but the circularity is the more fundamental threat to the central claim. The proposed concrete test — re-labeling a sample with independent ground truth, or using pre-deployment/non-gateway traces — would settle whether the labels are contaminated. Because this is an addressable empirical weakness rather than a demonstrated logical contradiction, the correct verdict remains CONDITIONAL: the architecture and deployment appear real, but the headline accuracy numbers require the independence check before they can be fully trusted. I therefore recommend UNCHANGED relative to the reader's CONDITIONAL verdict, with the condition sharpened to require disclosure of trace ground-truth provenance or an independent relabeling study.","tokens_in":27265,"tokens_out":7867,"duration_ms":90444,"concrete_test":"Recompute Recall@15 on a sample of ~200 queries per production trace using ground-truth labels obtained independently of the gateway: either (a) queries logged before Tool Search was enabled or from clients that mount the full tool list directly, or (b) human annotators selecting the tool warranted by the raw user query from the full 3,616-tool catalog. Report Hybrid vs dense-only recall on this independent set. If Hybrid recall falls below 98% or the gap to dense-only narrows to the ToolBench-like ~2 pp (97.5% vs 95.5% at N=640), the production numbers are contaminated by the agent's reliance on Tool Search, as stated in §7 Lesson 4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim — 98.2% Recall@15 at N=3,616 (§6.3) — is scored against 'ground truth in candidates' using four production traces (§6.1). The paper nowhere establishes that the labeled 'actually invoked' tool is independent of the gateway's Tool Search. In fact, §7 (Lesson 4) states that in the traced production deployment 'agents rely on tool recommendation for tool discovery during task execution,' and §6.2 describes the evaluation loop as 'the agent selects tools from the recommendation list (Top-15, 30, 60), and then issues tool calls through the gateway.' If the invoked tool is chosen from the recommended list, then Recall@15 measures the system agreeing with its own output: even a wrong tool recommended by the gateway and subsequently invoked counts as a hit, and a task that fails because the right tool was never retrieved still contributes a label from the candidate set. The same contamination would inflate the hybrid-over-dense advantage (98.2% vs 83.1% at N=3,616), since both are computed on the same system-influenced labels. This is the most load-bearing concern because it directly undermines the headline scalability and accuracy numbers; the architecture and deployment are otherwise plausible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a production MCP gateway deployed at Alibaba Cloud that (1) adapts legacy OpenAPI services to MCP and bridges incompatible MCP transports, (2) offloads authentication and fine-grained access control, (3) performs hybrid semantic/lexical tool recommendation with a Tool Search MCP tool, and (4) provides session-aware routing across replicated gateway instances using a centralized session store and, in an evolving design, Session-ID encoding. The central claims are that hybrid retrieval sustains 98.2% Recall@15 at 3,616 tools with bounded latency (<250 ms), that retrieve-then-select reduces end-to-end tool-selection time by 8.9× (55.66 s→6.24 s) and token usage by 23.8× (505.9k→21.2k tokens) versus inlining all tools, that protocol conversion overhead is sub-143 μs P50, and that session-aware routing adds meaningful but bounded overhead (P99 55.6 ms at 4 instances, 85% CPU at ~100 QPS). The paper also reports lessons from eight months of production operation, including long-lived connection benefits, knowledge/LRU caches, and automated query-rewrite rule extraction (RE-Agent).","tokens_in":27515,"tokens_out":2097,"duration_ms":25982,"significance":"If the claims hold, this is a valuable systems contribution: it is the first detailed description of a cloud-scale MCP gateway that unifies legacy API access, deterministic tool recommendation, and session-aware routing, with an actual eight-month production deployment. The measured 8.9×/23.8× latency/token reductions are directly relevant to the agent-infrastructure community, and the design is sufficiently concrete (protocol stages, RRF fusion, centralized session store, Pub/Sub response relay) to serve as a reusable template. The paper also makes a useful methodological point by comparing against public ToolBench and showing that fine-tuned retriever advantage does not transfer to cloud catalogs. Machine-checked proofs are not applicable, but the paper does report reproducible trace-based measurements and clearly discloses overheads and trade-offs (e.g., stateful routing P99 and CPU). The main risk to significance is the ground-truth independence of the production traces, discussed below.","major_comments":[{"comment":"The central Recall@15 claim may be partially self-confirmatory. §6.2 states the evaluation loop as 'the agent selects tools from the recommendation list (Top-15, 30, 60), and then issues tool calls through the gateway,' and §7 (Lesson 4) says 'agents rely on tool recommendation for tool discovery during task execution.' If the ground-truth 'actually invoked tool' in the four production traces was chosen from the gateway's recommendation list, then Recall@15 measures agreement of the retriever with its own downstream output rather than with an independent ground truth. A failed retrieval followed by a recovered call, or a wrong tool invoked from the candidate set, would still count as a hit. The paper must either state that trace labels come from direct non-gateway invocations before recommendation was in the loop, or provide a separate held-out evaluation with independently labeled tool","section":"§6.1, §6.3, §7 (Lesson 4)"},{"comment":"The ToolBench comparison has limited discriminative power for the hybrid approach. On ToolBench, Hybrid vs RAG-MCP is 97.5% vs 95.5% at N=640, which is a small gap, and both are well below ToolLLM's near-perfect recall because ToolLLM was fine-tuned on that corpus. The paper argues hybrid does not hurt on generic descriptions, but this does not establish that hybrid's lexical path helps when descriptions are generic. Since the paper's motivation for hybrid is that cloud tool names have lexical signal, and the production traces are the only evidence of that, the independence issue in the first comment also weakens the transferability conclusion. A synthetic or semi-controlled experiment varying tool-name informativeness would strengthen the claim that hybrid's advantage is structural rather than an artifact of trace labeling.","section":"§6.3, Figure 9c/9d"},{"comment":"The stateful gateway's overhead numbers are honestly reported, but the paper should reconcile the 85.3% CPU at ~100 QPS single-instance with the elastic-scale claim in Figure 15. Figure 15 shows per-instance CPU in the 20–30% range during production, while Figure 14a shows 85.3% at ~100 QPS in the controlled test. The discrepancy suggests either different hardware/workload characteristics or that the controlled test used a worst-case configuration. The paper should state which configuration (e.g., number of sessions, store RTT, cache hit rate) was used in Figure 14a so readers can judge whether the §6.4 overhead numbers represent the production operating point. This is not a correctness issue but matters for the 'stable under scale-out' claim.","section":"§6.4, Figure 14a"},{"comment":"The paper describes the deployment as 'validated on O(10^4) OpenAPI operations across 315 Alibaba Cloud products,' and the maximum per-user tool mount count reaches O(10^4). However, the headline scalability evaluation is performed on at most 3,616 tools. The paper should clarify whether any evaluation (even a subset) covers O(10^4) tools, or whether the 98% recall and <250 ms latency claims are only established for ≤3,616 tools. If the latter, the claim 'scales agent tool access to 3,000+' is accurate but should not be conflated with the O(10^4) supported catalog size.","section":"§5, §6.1"}],"minor_comments":[{"comment":"The abstract and intro claim '98% Top-15 recall' and '3,000+ tools' without stating the ground-truth independence caveat; the paper body should add one sentence specifying how ground truth was obtained.","section":"Abstract / §1"},{"comment":"The baseline in Table 3 uses 372 tools while the retrieval conditions use up to 3,616 tools. The paper states this is because the baseline cannot mount O(1k) tools, but readers may initially misread the comparison as same-tool-count. Add an explicit sentence that the baseline is evaluated at the largest tool count that fits in the context window (372) and that the retrieval pipeline is evaluated at 3,616; the comparison is therefore conservative in favor of the baseline in terms of tool-count difficulty.","section":"§6.2, Table 3"},{"comment":"Figure 11 shows latency for Top-K up to 1000 but the text only states a 202–243 ms range. The figure's x-axis is 'Top-K' and the legend says K=5,10,15,...; it would be clearer to also report the P99 or a table of the exact K=15 latency separately, since Top-15 is the production operating point.","section":"§6.3, Figure 11"},{"comment":"The RRF description mentions 'a smoothing constant' but does not give its value. State the constant used in the implementation; this is a one-line fix.","section":"§4.3, Figure 7"},{"comment":"The paper presents two alternatives — centralized store plus Pub/Sub and Session-ID encoding — and notes the system is 'evolving toward' the latter. The evaluation (§6.4) is based on the centralized approach. Add a sentence clarifying that the overhead numbers in Figure 14a are for the centralized-deployment configuration, so readers do not assume the encoding approach is what was measured.","section":"§7, Lesson 1"},{"comment":"Equation (1) is sound but the notation 'Acc_select|recalled' is a bit unusual; consider defining it as the LLM selection accuracy conditioned on the ground-truth tool being in the candidate set.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong systems contribution with a real deployment. The central risk is the ground-truth independence of the production traces; this is a one-time fix (clarify labeling provenance, or add an independent-label evaluation). I do not see a case for rejection because the architectural contributions — protocol adaptation, session-aware routing, and the deployment lessons — stand even if the recall numbers turn out to be partially self-confirmatory. However, the headline '98.2% Recall@15' is the most cited number, so the authors must address it head-on. I also note that the paper does not claim a formal model or theorem; its value is empirical, so the evaluation provenance is what makes or breaks the main claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about agent-tool infrastructure. It's a real systems paper from Alibaba Cloud: an MCP gateway that has been in production for eight months, with a coherent architecture (protocol adaptation for legacy OpenAPI, hybrid dense+sparse retrieval with RRF, session-aware routing via a centralized store plus Pub/Sub bridging). The measurements are internally consistent—the 8.9x and 23.8x reductions check out against the tables—and the authors are unusually honest about their own costs: stateful routing hits 55.6 ms P99 at four instances and burns 85% CPU at ~100 QPS. That is worth reading on its own.\n\nNow the soft spot, and it is the load-bearing one. The stress-test concern lands. Recall@15 is scored against the tool actually invoked in the production traces, but Section 7 says agents rely on tool recommendation during task execution, and Section 6.2 describes the evaluation loop as the agent selecting tools from the recommendation list. So the ground-truth label is not independent of the system being evaluated. A wrong tool that the gateway recommends and the agent calls counts as a hit. That inflates the 98.2% recall and the hybrid-over-dense advantage, since both are computed on the same system-influenced labels. This does not touch the time and token cost reductions—those are valid regardless—but the accuracy claims are compromised until the authors clarify whether the traced invocations were made through the gateway's Tool Search or before it.\n\nMinor issues: no artifact release, no variance or confidence intervals on the headline numbers, and the RE-Agent rewrite rules in Lesson 4 are extracted from the same tool-description corpus and evaluated on Trace-3, which smells like tuning on the test set.\n\nThe architecture and deployment are otherwise plausible, and the lessons (long-lived connections, the knowledge cache, session-ID encoding) are genuinely useful. This is the kind of paper that should go to peer review and get a chance to answer the circularity question. I would accept with major revision, and a serious referee can fix the evaluation narrative without throwing out the system. Worth bringing to a reading group for the methodology debate, and worth citing if you work on MCP or agent-tool access.","headline":"A serious, deployed MCP gateway with honest cost numbers, but the headline recall is likely contaminated by ground-truth circularity; the time/token savings still stand.","tokens_in":28199,"tokens_out":2516,"would_cite":true,"duration_ms":31293,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agents need not swallow every tool description: a gateway that retrieves the ~15 relevant tools per query via hybrid lexical-semantic search sustains 98% Top-15 recall over 3,616 cloud tools, cutting selection time 8.9× and tokens 23.8×.","keywords":["Model Context Protocol (MCP)","LLM agents","tool recommendation","hybrid retrieval","session-aware routing","cloud gateway","retrieve-then-select","OpenAPI-to-MCP"],"falsifier":"Two checks settle it. Re-label the four production traces using only invocations that bypassed the gateway's own Tool Search and recompute Recall@15 on that subset — if recall drops materially below 98%, self-recommendation inflates the score. Or run the same hybrid-versus-dense comparison on a catalog with anonymized, natural-language-only tool names; the paper's benchmark data already shows the hinge (the hybrid edge shrinks from about 15 points to about 2 points there), so a zero-lexical-signal corpus tests whether the hybrid advantage survives at all.","tokens_in":27070,"feed_emoji":"⚙️","tokens_out":21457,"duration_ms":202574,"temperature":0.7,"pith_summary":"The paper sets out to answer a practical question: how can tool-using LLM agents operate over the thousands of APIs a cloud platform exposes, when even the largest frontier models effectively saturate at a few hundred tool descriptions and degrade as the list grows? Its answer is that the task of narrowing the tool set should be moved out of the LLM entirely and onto a gateway that sits between agents and tool servers, in the same position a web load balancer occupied in the previous era. The gateway translates legacy APIs into Model Context Protocol (MCP) tools — the emerging standard interface for agent tool calls — bridges incompatible protocol versions, enforces identity-based tool access, and, the load-bearing piece, runs a deterministic hybrid search that retrieves the roughly fifteen likely-relevant tools for each query. In an eight-month production deployment, that retrieve-then-select pipeline held 98% Top-15 recall as the catalog grew past 3,600 tools, kept recommendation latency under 250 ms, and cut tool-selection time 8.9× and token consumption 23.8× relative to mounting every tool, while a centralized session store and hash-based routing preserved stateful sessions across replicated gateways. If the claims hold, the context window stops being the ceiling on tool access, and the standard web-era pattern of gateway-plus-retrieval becomes the reusable shape of agent infrastructure.","feed_headline":"Retrieve 15 tools, not 3,616, and agents scale","feed_subtitle":"A gateway-side hybrid search holds 98% Top-15 recall while cutting agent tool-selection time 8.9x and tokens 23.8x.","key_machinery":"The central object is the gateway: a shared control point on the MCP data plane that terminates client connections and re-establishes them to backends. Its load-bearing component is hybrid tool retrieval — a dual-path index holding each tool text as both a dense embedding and a sparse BM25 vocabulary, fused by Reciprocal Rank Fusion (a sum of reciprocal ranks) into a top-15 list, exploiting cloud catalogs' property that tool names encode function so lexical match complements semantic search. Supporting mechanisms: a Tool Search meta-tool exposing discovery as an ordinary MCP tool; session-aware routing keeping frontend-to-backend mappings in a centralized store, forwarding a derived Routing-","core_discovery":"Central claim: exposing every tool schema to the LLM is the wrong scaling point; gateway-side retrieval carries the scale. Against a catalog growing from 394 to 3,616 production tools, hybrid dense-plus-BM25 retrieval holds 98.2% Recall@15 in 202–243 ms, where dense-only retrieval falls to 83.1%. Giving the LLM the retrieved 15 tools keeps end-to-end accuracy at 81.6% at the largest scale and cuts selection time from 55.7 s to 6.2 s and tokens from 505.9k to 21.2k. The gateway also compiles legacy OpenAPI (web-API) operations into MCP tools at negligible cost and preserves stateful sessions across replicated instances via a centralized session store, hash-based routing, and a Pub/Sub bridge","pith_inferences":["The reported 15-point recall edge over dense-only search on production tools shrinks to about 2 points (97.5% vs 95.5%) on the paper's benchmark corpus with generic descriptions. My reading: the flagship gains are specific to catalogs whose tool names are dense identifiers; catalogs described in ordinary prose should expect a smaller lexical lift, and providers who want this behavior should keep n","The gateway's parse-annotate-rewrite-forward pipeline is a general answer to a structural fact: any agent protocol that keeps routing-relevant signals in message bodies defeats conventional header-based load balancers. I would expect the same template to serve future tool protocols if body-carried routing becomes the norm.","The knowledge-cache idea — extract prerequisite chains statically from foreign-key relationships — is described for cloud resource schemas, but it should transfer to any tool corpus with typed parameters (database operations, workflow engines, CI pipelines). A quick test: inject schema-derived chains into retrieval for such a corpus and measure whether end-to-end task success stops decaying with c","If lexical name matching is carrying recall, then tool-name hygiene is an operational lever, not a cosmetic one: near-duplicate or generic names would silently depress recall as catalogs grow. One consequence I would draw: tool-naming conventions deserve the same standardization attention as the transport protocol."],"forward_implications":["Cloud-scale tool access becomes feasible with existing models: with Top-15 pre-filtering, end-to-end selection accuracy holds at 81.6% over 3,616 tools, whereas inlining the same catalog exceeds every tested model's context window (fails at 752+ tools).","Selection cost becomes bounded and predictable: gateway-side retrieval finishes in 202–243 ms, replacing up to 55.7 s of LLM selection time (8.9×) and 505.9k tokens (23.8×), so cost stops growing with catalog size.","The legacy web-API estate — on the order of 10^4 operations at the platform where this was deployed — can be mounted as MCP tools without backend refactoring: each API operation compiles to one MCP tool at mount time, with median per-request conversion cost at most 143 μs.","Replicated, stateful MCP backends remain correct as the gateway scales out: session mappings live in a centralized store, and long-lived gateway-backend connections cut mean/P90/P99 latency by 51.7%/55.7%/56.5%.","Multi-step tool dependencies can be resolved in one retrieval: schema-derived prerequisite chains complete 100% of queries with zero extra LLM round-trips, while iterative discovery collapses to 0% on chains of length five or more."],"fun_headline_variants":["Gateway retrieval picks 15 tools, agents scale to 3,000+","Hybrid search holds 98% recall, cuts tokens 23.8x","MCP gateway retrieves 15 tools, not 3,616, keeps 81.6% accuracy","Tool selection time down 8.9x and tokens 23.8x via hybrid retrieval","Scaling agent tools to 3,000+ while picking only 15 per call"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The production numbers assume the trace labels record the tool a user independently needed rather than one the gateway itself recommended, and that cloud tool names carry their meaning in their identifiers; if either fails, the 98% recall partly measures the system agreeing with its own suggestions or an artifact of one catalog's naming style.","fun_headline_variants_meta":{"raw":{"variants":["Gateway retrieval picks 15 tools, agents scale to 3,000+","Hybrid search holds 98% recall, cuts tokens 23.8x","MCP gateway retrieves 15 tools, not 3,616, keeps 81.6% accuracy","Tool selection time down 8.9x and tokens 23.8x via hybrid retrieval","Scaling agent tools to 3,000+ while picking only 15 per call"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":2999,"prompt_tokens":805,"completion_tokens":2194,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2089}},"tokens_in":549,"tokens_out":2194,"duration_ms":13995,"temperature":1.0,"reasoning_tokens":2089,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:50:18.468685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two checks settle it. Re-label the four production traces using only invocations that bypassed the gateway's own Tool Search and recompute Recall@15 on that subset — if recall drops materially below 98%, self-recommendation inflates the score. Or run the same hybrid-versus-dense comparison on a catalog with anonymized, natural-language-only tool names; the paper's benchmark data already shows the hinge (the hybrid edge shrinks from about 15 points to about 2 points there), so a zero-lexical-signal corpus tests whether the hybrid advantage survives at all.","supporting_citations":[],"review_version":1}