Pith. sign in

REVIEW 4 major objections 6 minor 80 references

Agents need not swallow every tool description: a gateway that retrieves the ~15 relevant tools per query via hybrid lexical-semantic search sustains 98% Top-15 recall over 3,616 cloud tools, cutting selection time 8.9× and tokens 23.8×.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:50 UTC pith:X5QOURSX

load-bearing objection A serious, deployed MCP gateway with honest cost numbers, but the headline recall is likely contaminated by ground-truth circularity; the time/token savings still stand. the 4 major comments →

arxiv 2607.15593 v1 pith:X5QOURSX submitted 2026-07-17 cs.DC cs.AIcs.NI

Scalable LLM Agent Tool Access in the Cloud

Mingxin Li , Enge Song , Yueshang Zuo , Xiaodong Liu , Rong Wen , Qiang Fu , Gianni Antichi , Jian He
show 22 more authors
Jing Tie Zhou Shao Xiaobo Xue Xiong Xiao Luyao Zhong Shaokai Zhang Jiangu Zhao Jianyuan Lu Shize Zhang Xiaoqing Sun Changgang Zheng Zihao Fan Haonan Li Tian Pan Xiaomin Wu Yang Song Xing Li Biao Lyu Meng Li Haipeng Dai Guihai Chen Shunmin Zhu
This is my paper
classification cs.DC cs.AIcs.NI
keywords Model Context Protocol (MCP)LLM agentstool recommendationhybrid retrievalsession-aware routingcloud gatewayretrieve-then-selectOpenAPI-to-MCP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to answer a practical question: how can tool-using LLM agents operate over the thousands of APIs a cloud platform exposes, when even the largest frontier models effectively saturate at a few hundred tool descriptions and degrade as the list grows? Its answer is that the task of narrowing the tool set should be moved out of the LLM entirely and onto a gateway that sits between agents and tool servers, in the same position a web load balancer occupied in the previous era. The gateway translates legacy APIs into Model Context Protocol (MCP) tools — the emerging standard interface for agent tool calls — bridges incompatible protocol versions, enforces identity-based tool access, and, the load-bearing piece, runs a deterministic hybrid search that retrieves the roughly fifteen likely-relevant tools for each query. In an eight-month production deployment, that retrieve-then-select pipeline held 98% Top-15 recall as the catalog grew past 3,600 tools, kept recommendation latency under 250 ms, and cut tool-selection time 8.9× and token consumption 23.8× relative to mounting every tool, while a centralized session store and hash-based routing preserved stateful sessions across replicated gateways. If the claims hold, the context window stops being the ceiling on tool access, and the standard web-era pattern of gateway-plus-retrieval becomes the reusable shape of agent infrastructure.

Core claim

Central claim: exposing every tool schema to the LLM is the wrong scaling point; gateway-side retrieval carries the scale. Against a catalog growing from 394 to 3,616 production tools, hybrid dense-plus-BM25 retrieval holds 98.2% Recall@15 in 202–243 ms, where dense-only retrieval falls to 83.1%. Giving the LLM the retrieved 15 tools keeps end-to-end accuracy at 81.6% at the largest scale and cuts selection time from 55.7 s to 6.2 s and tokens from 505.9k to 21.2k. The gateway also compiles legacy OpenAPI (web-API) operations into MCP tools at negligible cost and preserves stateful sessions across replicated instances via a centralized session store, hash-based routing, and a Pub/Sub bridge

What carries the argument

The central object is the gateway: a shared control point on the MCP data plane that terminates client connections and re-establishes them to backends. Its load-bearing component is hybrid tool retrieval — a dual-path index holding each tool text as both a dense embedding and a sparse BM25 vocabulary, fused by Reciprocal Rank Fusion (a sum of reciprocal ranks) into a top-15 list, exploiting cloud catalogs' property that tool names encode function so lexical match complements semantic search. Supporting mechanisms: a Tool Search meta-tool exposing discovery as an ordinary MCP tool; session-aware routing keeping frontend-to-backend mappings in a centralized store, forwarding a derived Routing-

Load-bearing premise

The production numbers assume the trace labels record the tool a user independently needed rather than one the gateway itself recommended, and that cloud tool names carry their meaning in their identifiers; if either fails, the 98% recall partly measures the system agreeing with its own suggestions or an artifact of one catalog's naming style.

What would settle it

Two checks settle it. Re-label the four production traces using only invocations that bypassed the gateway's own Tool Search and recompute Recall@15 on that subset — if recall drops materially below 98%, self-recommendation inflates the score. Or run the same hybrid-versus-dense comparison on a catalog with anonymized, natural-language-only tool names; the paper's benchmark data already shows the hinge (the hybrid edge shrinks from about 15 points to about 2 points there), so a zero-lexical-signal corpus tests whether the hybrid advantage survives at all.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Cloud-scale tool access becomes feasible with existing models: with Top-15 pre-filtering, end-to-end selection accuracy holds at 81.6% over 3,616 tools, whereas inlining the same catalog exceeds every tested model's context window (fails at 752+ tools).
  • Selection cost becomes bounded and predictable: gateway-side retrieval finishes in 202–243 ms, replacing up to 55.7 s of LLM selection time (8.9×) and 505.9k tokens (23.8×), so cost stops growing with catalog size.
  • The legacy web-API estate — on the order of 10^4 operations at the platform where this was deployed — can be mounted as MCP tools without backend refactoring: each API operation compiles to one MCP tool at mount time, with median per-request conversion cost at most 143 μs.
  • Replicated, stateful MCP backends remain correct as the gateway scales out: session mappings live in a centralized store, and long-lived gateway-backend connections cut mean/P90/P99 latency by 51.7%/55.7%/56.5%.
  • Multi-step tool dependencies can be resolved in one retrieval: schema-derived prerequisite chains complete 100% of queries with zero extra LLM round-trips, while iterative discovery collapses to 0% on chains of length five or more.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported 15-point recall edge over dense-only search on production tools shrinks to about 2 points (97.5% vs 95.5%) on the paper's benchmark corpus with generic descriptions. My reading: the flagship gains are specific to catalogs whose tool names are dense identifiers; catalogs described in ordinary prose should expect a smaller lexical lift, and providers who want this behavior should keep n
  • The gateway's parse-annotate-rewrite-forward pipeline is a general answer to a structural fact: any agent protocol that keeps routing-relevant signals in message bodies defeats conventional header-based load balancers. I would expect the same template to serve future tool protocols if body-carried routing becomes the norm.
  • The knowledge-cache idea — extract prerequisite chains statically from foreign-key relationships — is described for cloud resource schemas, but it should transfer to any tool corpus with typed parameters (database operations, workflow engines, CI pipelines). A quick test: inject schema-derived chains into retrieval for such a corpus and measure whether end-to-end task success stops decaying with c
  • If lexical name matching is carrying recall, then tool-name hygiene is an operational lever, not a cosmetic one: near-duplicate or generic names would silently depress recall as catalogs grow. One consequence I would draw: tool-naming conventions deserve the same standardization attention as the transport protocol.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a production MCP gateway deployed at Alibaba Cloud that (1) adapts legacy OpenAPI services to MCP and bridges incompatible MCP transports, (2) offloads authentication and fine-grained access control, (3) performs hybrid semantic/lexical tool recommendation with a Tool Search MCP tool, and (4) provides session-aware routing across replicated gateway instances using a centralized session store and, in an evolving design, Session-ID encoding. The central claims are that hybrid retrieval sustains 98.2% Recall@15 at 3,616 tools with bounded latency (<250 ms), that retrieve-then-select reduces end-to-end tool-selection time by 8.9× (55.66 s→6.24 s) and token usage by 23.8× (505.9k→21.2k tokens) versus inlining all tools, that protocol conversion overhead is sub-143 μs P50, and that session-aware routing adds meaningful but bounded overhead (P99 55.6 ms at 4 instances, 85% CPU at ~100 QPS). The paper also reports lessons from eight months of production operation, including long-lived connection benefits, knowledge/LRU caches, and automated query-rewrite rule extraction (RE-Agent).

Significance. If the claims hold, this is a valuable systems contribution: it is the first detailed description of a cloud-scale MCP gateway that unifies legacy API access, deterministic tool recommendation, and session-aware routing, with an actual eight-month production deployment. The measured 8.9×/23.8× latency/token reductions are directly relevant to the agent-infrastructure community, and the design is sufficiently concrete (protocol stages, RRF fusion, centralized session store, Pub/Sub response relay) to serve as a reusable template. The paper also makes a useful methodological point by comparing against public ToolBench and showing that fine-tuned retriever advantage does not transfer to cloud catalogs. Machine-checked proofs are not applicable, but the paper does report reproducible trace-based measurements and clearly discloses overheads and trade-offs (e.g., stateful routing P99 and CPU). The main risk to significance is the ground-truth independence of the production traces, discussed below.

major comments (4)
  1. [§6.1, §6.3, §7 (Lesson 4)] The central Recall@15 claim may be partially self-confirmatory. §6.2 states the evaluation loop as 'the agent selects tools from the recommendation list (Top-15, 30, 60), and then issues tool calls through the gateway,' and §7 (Lesson 4) says 'agents rely on tool recommendation for tool discovery during task execution.' If the ground-truth 'actually invoked tool' in the four production traces was chosen from the gateway's recommendation list, then Recall@15 measures agreement of the retriever with its own downstream output rather than with an independent ground truth. A failed retrieval followed by a recovered call, or a wrong tool invoked from the candidate set, would still count as a hit. The paper must either state that trace labels come from direct non-gateway invocations before recommendation was in the loop, or provide a separate held-out evaluation with independently labeled tool
  2. [§6.3, Figure 9c/9d] The ToolBench comparison has limited discriminative power for the hybrid approach. On ToolBench, Hybrid vs RAG-MCP is 97.5% vs 95.5% at N=640, which is a small gap, and both are well below ToolLLM's near-perfect recall because ToolLLM was fine-tuned on that corpus. The paper argues hybrid does not hurt on generic descriptions, but this does not establish that hybrid's lexical path helps when descriptions are generic. Since the paper's motivation for hybrid is that cloud tool names have lexical signal, and the production traces are the only evidence of that, the independence issue in the first comment also weakens the transferability conclusion. A synthetic or semi-controlled experiment varying tool-name informativeness would strengthen the claim that hybrid's advantage is structural rather than an artifact of trace labeling.
  3. [§6.4, Figure 14a] The stateful gateway's overhead numbers are honestly reported, but the paper should reconcile the 85.3% CPU at ~100 QPS single-instance with the elastic-scale claim in Figure 15. Figure 15 shows per-instance CPU in the 20–30% range during production, while Figure 14a shows 85.3% at ~100 QPS in the controlled test. The discrepancy suggests either different hardware/workload characteristics or that the controlled test used a worst-case configuration. The paper should state which configuration (e.g., number of sessions, store RTT, cache hit rate) was used in Figure 14a so readers can judge whether the §6.4 overhead numbers represent the production operating point. This is not a correctness issue but matters for the 'stable under scale-out' claim.
  4. [§5, §6.1] The paper describes the deployment as 'validated on O(10^4) OpenAPI operations across 315 Alibaba Cloud products,' and the maximum per-user tool mount count reaches O(10^4). However, the headline scalability evaluation is performed on at most 3,616 tools. The paper should clarify whether any evaluation (even a subset) covers O(10^4) tools, or whether the 98% recall and <250 ms latency claims are only established for ≤3,616 tools. If the latter, the claim 'scales agent tool access to 3,000+' is accurate but should not be conflated with the O(10^4) supported catalog size.
minor comments (6)
  1. [Abstract / §1] The abstract and intro claim '98% Top-15 recall' and '3,000+ tools' without stating the ground-truth independence caveat; the paper body should add one sentence specifying how ground truth was obtained.
  2. [§6.2, Table 3] The baseline in Table 3 uses 372 tools while the retrieval conditions use up to 3,616 tools. The paper states this is because the baseline cannot mount O(1k) tools, but readers may initially misread the comparison as same-tool-count. Add an explicit sentence that the baseline is evaluated at the largest tool count that fits in the context window (372) and that the retrieval pipeline is evaluated at 3,616; the comparison is therefore conservative in favor of the baseline in terms of tool-count difficulty.
  3. [§6.3, Figure 11] Figure 11 shows latency for Top-K up to 1000 but the text only states a 202–243 ms range. The figure's x-axis is 'Top-K' and the legend says K=5,10,15,...; it would be clearer to also report the P99 or a table of the exact K=15 latency separately, since Top-15 is the production operating point.
  4. [§4.3, Figure 7] The RRF description mentions 'a smoothing constant' but does not give its value. State the constant used in the implementation; this is a one-line fix.
  5. [§7, Lesson 1] The paper presents two alternatives — centralized store plus Pub/Sub and Session-ID encoding — and notes the system is 'evolving toward' the latter. The evaluation (§6.4) is based on the centralized approach. Add a sentence clarifying that the overhead numbers in Figure 14a are for the centralized-deployment configuration, so readers do not assume the encoding approach is what was measured.
  6. [Appendix E] Equation (1) is sound but the notation 'Acc_select|recalled' is a bit unusual; consider defining it as the LLM selection accuracy conditioned on the ground-truth tool being in the candidate set.

Circularity Check

1 steps flagged

Recall@15 ground truth is generated by the same recommendation pipeline being evaluated, making the headline recall partially self-confirmatory.

specific steps
  1. self definitional [§6.1 Protocol and Metrics; §6.2 Agent Tool Selection Performance; §7 Lesson 4; Figure 9(a)/Figure 12]
    "The retriever returns Top-15, from which the LLM selects one. We report Recall@15 (ground truth in candidates...) ... For each request, the agent selects tools from the recommendation list (Top-15, 30, 60), and then issues tool calls through the gateway. ... agents rely on tool recommendation for tool discovery during task execution."

    The ground-truth label for Recall@15 is 'the tool actually invoked' in production traces. But the same traces are produced by agents whose tool discovery goes through the gateway's Tool Search, and the evaluation loop restricts the agent to invoke tools from the recommended list. Thus any tool not recommended cannot be invoked and never becomes a positive label, while an incorrectly recommended tool that gets invoked counts as a hit. Recall@15/selection accuracy therefore measures the system agreeing with its own output, not retrieval against an independent 'right tool.' The 98.2% figure is partly forced by the label-generation mechanism, and the hybrid-vs-dense comparison inherits the same bias.

full rationale

The paper is a deployed-systems measurement paper, not a derivation from equations; most claims (protocol conversion latency, session-routing overhead, scale-out behavior) are measured against absolute production numbers or external baselines and are not circular. The one load-bearing circularity is the evaluation of tool recommendation: the Recall@15 ground truth is not shown to be independent of the gateway's own Tool Search, and the paper's own §7 states that agents rely on tool recommendation for tool discovery during the traced period, while §6.2 describes an evaluation loop in which the agent may only invoke tools from the recommended list. Under those conditions, the headline 98.2% Recall@15 is partly self-consistent by construction. The self-citation to Hermes [41] is background and not load-bearing. The RE-Agent rule extraction is a corpus-derived feature and not circular by itself, though its recall numbers inherit the same label-contamination concern.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

No new protocol-level entities, forces, or conserved quantities are postulated. Design artifacts (Tool Search meta-tool, Session-ID encoding, Knowledge Cache) are software constructs within the paper's own system with no independent falsifiable handle outside the paper; the Tool Search interface is the closest to an invented interface, but it is fully specified and testable by any MCP client.

free parameters (3)
  • Retrieval candidate count K = 15 (swept over {5,10,15,20,25,30,40,50}, Appendix E)
    Headline 'Top-15 recall' is reported at the K chosen by sweeping the same datasets on which it is evaluated (§6.3, Appendix E). The paper also reports K=10/20/25/30/60 in supporting figures.
  • Query-rewrite entity-pattern rules (RE-Agent) = auto-extracted from Alibaba Cloud tool descriptions
    The rewrite rules mapping identifier patterns to entity types are learned from the same domain corpus on which recall is then measured (Figures 20-21, Trace-3); they encode domain knowledge into the query and push Top-15 recall from 80.8% to 99.7%.
  • LRU semantic cache size = e.g., 64 entries
    Chosen cache size supporting the up-to-60% latency reduction claim (§7, Lesson 3); does not affect the central accuracy claims.
axioms (6)
  • domain assumption Ground-truth tool invocations in the production traces are externally determined, independent of the gateway's own recommendations.
    Underlies Recall@15 and selection-accuracy measurement (§6.1, §6.3). If trace agents used the same Tool Search to pick tools, recall is partially self-referential.
  • domain assumption MCP tool names and cloud tool descriptions in the target catalogs encode functional keywords well enough that lexical matching (BM25) adds signal beyond embeddings.
    Stated in §3.3 ('we exploit the fact that MCP tool names often explicitly encode functionality'). The hybrid advantage over dense-only retrieval is small on ToolBench (97.5 vs 95.5), so the flagship recall numbers depend on this naming convention.
  • domain assumption OpenAPI operations can be mapped one-to-one to MCP tools without loss of selectability.
    §3.1/§4.1: 'Each OpenAPI operation is compiled into exactly one MCP tool.' The selection-accuracy results assume this granularity is the right one.
  • domain assumption MCP's de facto status and its transport/version churn continue as observed (HTTP+SSE vs Streamable HTTP, fragmented auth).
    Motivates the whole gateway architecture (§2.2-2.3). If the protocol consolidates, parts of the compatibility layer lose value.
  • standard math Standard IR machinery (ANN search, BM25, Reciprocal Rank Fusion) behaves as claimed.
    Used in §4.3 and not re-derived in the paper.
  • domain assumption The pre-trained embedding encoder yields query/tool representations adequate for the production cloud domain.
    The dense path of the hybrid index relies on this (§4.3). The paper itself documents the failure mode for ID-only queries (§7, Lesson 4).

pith-pipeline@v1.3.0-alltime-deepseek · 26816 in / 16875 out tokens · 199127 ms · 2026-08-01T22:50:18.468685+00:00 · methodology

0 comments
read the original abstract

LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. Operating MCP at cloud scale, however, becomes difficult. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tool is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy, and reduces tool selection time by $8.9\times$ and token usage by $23.8\times$, with low per-call overhead, stable under scale-out. Finally, we share the lessons learned from deploying the gateway system in production.

Figures

Figures reproduced from arXiv: 2607.15593 by Biao Lyu, Changgang Zheng, Enge Song, Gianni Antichi, Guihai Chen, Haipeng Dai, Haonan Li, Jiangu Zhao, Jian He, Jianyuan Lu, Jing Tie, Luyao Zhong, Meng Li, Mingxin Li, Qiang Fu, Rong Wen, Shaokai Zhang, Shize Zhang, Shunmin Zhu, Tian Pan, Xiaobo Xue, XiaoDong Liu, Xiaomin Wu, Xiaoqing Sun, Xing Li, Xiong Xiao, Yang Song, Yueshang Zuo, Zhou Shao, Zihao Fan.

Figure 1
Figure 1. Figure 1: MCP 25/12 26/01 26/02 26/03 26/04 # Instances n × 103 Instances Requests # Requests n × 108 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: A concise MCP interaction workflow 69]. Unlike early chatbots that focused on prompt-based text generation without external interaction, LLM agents can plan, use tools, and accomplish complex goals. Using reasoning-and-acting paradigms such as ReAct, CoT, and ToT, agents decompose user intents into multi-step tasks [61, 67, 68] and interact with external systems—for example, query￾ing databases, managing c… view at source ↗
Figure 4
Figure 4. Figure 4: The measured impact of tool scale. (a) # tools may exceed the capacity of models. (b) Inlining schemas incurs context overhead. (c) The scale of the tools leads to huge inference latency. (d) Tool-selection accuracy degrades. of distinct operations. Even a single service category can include hundreds of tools (Figure 4a), making it difficult for models to accommodate tools from multiple categories. For exa… view at source ↗
Figure 5
Figure 5. Figure 5: The gateway system architecture System overview. The system ( [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Fine-grained access-control function offloading Multi-stage request routing. MCP services use HTTP+SSE or Streamable HTTP, while legacy OpenAPI services are ex￾posed as MCP tools via protocol conversion. Upon receiving a request, the gateway resolves the execution target and transport through a staged routing pipeline (Figure A4) that parses the tool invocation request in both header and body, annotates it… view at source ↗
Figure 7
Figure 7. Figure 7: Hybrid tool retrieval engine execution; the host then invokes tools through the standard LLM loop over this reduced set. Interface for tool recommendation. To realize agent￾driven discovery, the recommendation capability must be invocable from the agent tool-calling loop. The original MCP List Tools does not support query- or constraint-based input as retrieval parameters, limiting the functionality of too… view at source ↗
Figure 9
Figure 9. Figure 9: Retrieval comparison across methods on production cloud tools (a–b) and ToolBench (c–d), Qwen3.6-plus 50 60 70 Request Index 10 20 Time Cost (s) Top-15 Top-30 Top-60 Baseline (a) Time cost 480 500 Request Index 0 50 100 Token Usage (k) Top-15 Top-30 Top-60 Baseline (b) Token usage [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 13
Figure 13. Figure 13: API conversion overhead: online request conver￾sion (a, b) and offline tool generation (c). Comparison with retrieval baselines [PITH_FULL_IMAGE:figures/full_fig_p009_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Gateway RT breakdown vs. # instances 19:30 20:00 20:30 21:00 21:30 22:00 22:30 23:00 QPS n × 10 5 Scale Scale 20 30 CPU (%) (a) QPS & CPU usage 19:30 20:00 20:30 21:00 21:30 22:00 22:30 23:00 QPS n × 10 5 Scale Scale 0 50 100 RT (ms) (b) QPS & RT [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗
Figure 17
Figure 17. Figure 17: Latency w/o and with long-lived connection Query Knowledge Evict F D E B C A = Pair <Intent, Tool Set> Set <Tool DAG> Retriever Hybrid Retrieval Engine 2 Tools Cache Hit LRU Cache 1 4 Cache Miss 1 2 Top-K 3 DAG 5 Tools Insert [PITH_FULL_IMAGE:figures/full_fig_p011_17.png] view at source ↗
Figure 20
Figure 20. Figure 20: Rewrite-Agent and RE-Agent workflow O0 O1 O2 Auto 0.800 1.000 Top-15 Recall 0.808 0.824 0.995 0.997 [PITH_FULL_IMAGE:figures/full_fig_p012_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 16 linked inside Pith

  1. [1]

    2026.AgentCore Gateway.https://docs.a ws.amazon.com/bedrock-agentcore/latest/devguide/gateway.html Accessed: 2026-01-06

    AWS Bedrock AgentCore. 2026.AgentCore Gateway.https://docs.a ws.amazon.com/bedrock-agentcore/latest/devguide/gateway.html Accessed: 2026-01-06

    2026

  2. [2]

    2026.agentgateway: Next Generation Agentic Proxy for AI Agents and MCP servers.https://agentgateway.dev/ Accessed: 2026-06-10

    agentgateway authors. 2026.agentgateway: Next Generation Agentic Proxy for AI Agents and MCP servers.https://agentgateway.dev/ Accessed: 2026-06-10

    2026

  3. [3]

    2025.Alibaba Cloud API Documentation.https: //api.alibabacloud.com/documentAccessed: 2025-12-17

    Alibaba Cloud. 2025.Alibaba Cloud API Documentation.https: //api.alibabacloud.com/documentAccessed: 2025-12-17

    2025

  4. [4]

    2025.Claude Opus 4.5.https://www.anthropic.com/news /claude-opus-4-5Accessed: 2026-01-06

    Anthropic. 2025.Claude Opus 4.5.https://www.anthropic.com/news /claude-opus-4-5Accessed: 2026-01-06

    2025

  5. [5]

    2025.Introducing Claude Opus 4.5.https://www.anthropic

    Anthropic. 2025.Introducing Claude Opus 4.5.https://www.anthropic. com/news/claude-opus-4-5Accessed: 2026-01-06

    2025

  6. [6]

    2026.Introducing the Model Context Protocol.https: //www.anthropic.com/news/model-context-protocolAccessed: 2026-01-06

    Anthropic. 2026.Introducing the Model Context Protocol.https: //www.anthropic.com/news/model-context-protocolAccessed: 2026-01-06

    2026

  7. [7]

    2026.Apache APISIX Documentation.https: //apisix.apache.org/docs/apisix/getting-started/README/Accessed: 2026-01-06

    Apache APISIX Authors. 2026.Apache APISIX Documentation.https: //apisix.apache.org/docs/apisix/getting-started/README/Accessed: 2026-01-06

    2026

  8. [8]

    2026.Envoy Documentation.https://www.en voyproxy.io/docs/envoy/latest/Accessed: 2026-01-06

    Envoy Project Authors. 2026.Envoy Documentation.https://www.en voyproxy.io/docs/envoy/latest/Accessed: 2026-01-06

    2026

  9. [9]

    2026.HAProxy Documentation.https: //docs.haproxy.org/Accessed: 2026-01-06

    HAProxy Project Authors. 2026.HAProxy Documentation.https: //docs.haproxy.org/Accessed: 2026-01-06

    2026

  10. [10]

    2026.NGINX Documentation.https://nginx

    NGINX Project Authors. 2026.NGINX Documentation.https://nginx. org/en/docs/index.htmlAccessed: 2026-01-06

    2026

  11. [11]

    Zhuohang Bian, Feiyang Wu, Teng Ma, and Youwei Zhuo. 2025. Token- cake: A KV-Cache-centric Serving Framework for LLM-based Multi- Agent Applications.arXiv preprint arXiv:2510.18586(2025)

    Pith/arXiv arXiv 2025

  12. [12]

    Thierry Blankenstein, Jialin Yu, Zixuan Li, Vassilis Plachouras, Sunando Sengupta, Philip Torr, Yarin Gal, Alasdair Paren, and Adel Bibi. 2025. BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models.arXiv preprint arXiv:2510.00307(2025)

    arXiv 2025

  13. [13]

    Alan Chan, Kevin Wei, Sihao Huang, Nitarshan Rajkumar, Elija Per- rier, Seth Lazar, Gillian K Hadfield, and Markus Anderljung. 2025. Infrastructure for ai agents.arXiv preprint arXiv:2501.10114(2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Jinyuan Chen, Jiuchen Shi, Quan Chen, and Minyi Guo. 2025. Kairos: Low-latency multi-agent serving with shared llms and excessive loads in the public cloud.arXiv preprint arXiv:2508.06948(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    2025.LangChain: Building applications with LLMs through composability.https://www.langchain.comAccessed: 2025-11-19

    LangChain Contributors. 2025.LangChain: Building applications with LLMs through composability.https://www.langchain.comAccessed: 2025-11-19

    2025

  16. [16]

    2026.MCP Specification (Protocol Revision: 2024-11-05).https://modelcontextprotocol.io/specification/ 2024-11-05Accessed: 2026-01-06

    Model Context Protocol Contributors. 2026.MCP Specification (Protocol Revision: 2024-11-05).https://modelcontextprotocol.io/specification/ 2024-11-05Accessed: 2026-01-06

    2026

  17. [17]

    2026.MCP Hub.https://github.com/ravitem er/mcp-hubAccessed: 2026-01-06

    MCP Hub Contributors. 2026.MCP Hub.https://github.com/ravitem er/mcp-hubAccessed: 2026-01-06

    2026

  18. [18]

    2026.DeepSeek API Docs.https://api-docs.deepseek.com/ quick_start/pricingAccessed: 2026-01-06

    DeepSeek. 2026.DeepSeek API Docs.https://api-docs.deepseek.com/ quick_start/pricingAccessed: 2026-01-06

    2026

  19. [19]

    Meihao Fan, Ju Fan, Nan Tang, Lei Cao, Guoliang Li, and Xiaoyong Du

  20. [20]

    Tiantian Gan and Qiyao Sun. 2025. Rag-mcp: Mitigating prompt bloat in llm tool selection via retrieval-augmented generation.arXiv preprint arXiv:2505.03275(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, et al

  22. [22]

    Dawei Gao, Zitao Li, Yuexiang Xie, Weirui Kuang, Liuyi Yao, Bingchen Qian, Zhijian Ma, Yue Cui, Haohao Luo, Shen Li, et al . 2025. AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications.arXiv preprint arXiv:2508.16279(2025)

    Pith/arXiv arXiv 2025

  23. [23]

    Agentscope: A flexible yet robust multi-agent platform.arXiv preprint arXiv:2402.14034(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. 2025. Pie: A programmable serving system for emerging llm applications. InPro- ceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 415–430

    2025

  25. [25]

    2026.Gemini API Documents.https://ai.google.dev/ gemini-api/docs/modelsAccessed: 2026-01-06

    Google Gemini. 2026.Gemini API Documents.https://ai.google.dev/ gemini-api/docs/modelsAccessed: 2026-01-06

    2026

  26. [26]

    Yongjian Guo, Puzhuo Liu, Wanlun Ma, Zehang Deng, Xiaogang Zhu, Peng Di, Xi Xiao, and Sheng Wen. 2025. Systematic analysis of mcp security.arXiv preprint arXiv:2508.12538(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Hechuan Guo, Yongle Hao, Yue Zhang, Minghui Xu, Peizhuo Lv, Jiezhi Chen, and Xiuzhen Cheng. 2025. A measurement study of model context protocol ecosystem.arXiv preprint arXiv:2509.25292(2025)

    arXiv 2025

  28. [28]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  29. [29]

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model context protocol (mcp): Landscape, security threats, and future re- search directions.arXiv preprint arXiv:2503.23278(2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Guoliang Li, Xuanhe Zhou, and Xinyang Zhao. 2024. Llm for data management.Proceedings of the VLDB Endowment17, 12 (2024), 4213– 4216

    2024

  31. [31]

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient serving of {LLM-based} applications with semantic variable. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 929–945

    2024

  32. [32]

    Marco Laju, Donghyun Son, Saurabh Agarwal, Nitin Kedia, Myungjin Lee, Jayanth Srinivasa, and Aditya Akella. 2026. Nalar: An agent serving framework.arXiv preprint arXiv:2601.05109(2026). 13

    arXiv 2026

  33. [33]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics12 (2024), 157–173

    2024

  34. [34]

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E Gonzalez, et al. 2025. Autellix: An efficient serving engine for llm agents as general programs.arXiv preprint arXiv:2502.13965(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Samuel Lin, Jiawei Zhou, and Minlan Yu. 2025. An LLM-based Agen- tic Framework for Accessible NetworkControl.ACM SIGMETRICS Performance Evaluation Review53, 2 (2025), 15–20

    2025

  36. [36]

    2026.MCP.so.https://mcp.so/Accessed: 2026-01-06

    MCP.so. 2026.MCP.so.https://mcp.so/Accessed: 2026-01-06

    2026

  37. [37]

    Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. 2024. Aios: Llm agent operating system.arXiv preprint arXiv:2403.16971 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    2026.MCP Market.https://mcpmarket.com/Accessed: 2026-01-06

    MCP Market. 2026.MCP Market.https://mcpmarket.com/Accessed: 2026-01-06

    2026

  39. [39]

    2025.What is the Model Context Protocol (MCP)?https://modelcontextprotocol.io/Accessed: 2025-12-17

    Model Context Protocol. 2025.What is the Model Context Protocol (MCP)?https://modelcontextprotocol.io/Accessed: 2025-12-17

    2025

  40. [40]

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, Ion Stoica, and Joseph E Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems.arXiv preprint arXiv:2310.08560(2023)

    Pith/arXiv arXiv 2023

  41. [41]

    2026.Microsoft MCP Gateway.https://microsoft.github.io/ mcp-gateway/Accessed: 2026-01-06

    Microsoft. 2026.Microsoft MCP Gateway.https://microsoft.github.io/ mcp-gateway/Accessed: 2026-01-06

    2026

  42. [42]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. Gorilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems37 (2024), 126544–126565

    2024

  43. [43]

    2026.Bailian MCP Market.https: //bailian.console.aliyun.com/cn-beijing/?tab=mcp#/mcp-market Accessed: 2026-01-06

    Alibaba Cloud Bailian Platform. 2026.Bailian MCP Market.https: //bailian.console.aliyun.com/cn-beijing/?tab=mcp#/mcp-market Accessed: 2026-01-06

    2026

  44. [44]

    Tian Pan, Enge Song, Yueshang Zuo, Shaokai Zhang, Yang Song, Jiangu Zhao, Wengang Hou, Jianyuan Lu, Xiaoqing Sun, Shize Zhang, et al. 2025. Hermes: Enhancing Layer-7 Cloud Load Balancers with Userspace-Directed I/O Event Notification. InProceedings of the ACM SIGCOMM 2025 Conference. 363–380

    2025

  45. [45]

    Zhangcheng Qiang, Weiqing Wang, and Kerry Taylor. 2024. Agent- OM: Leveraging LLM Agents for Ontology Matching.Proceedings of the VLDB Endowment18, 3 (2024), 516–529

    2024

  46. [46]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving{LLM} chatbot. In23rd USENIX Conference on File and Storage Technologies (FAST 25). 155–170

    2025

  47. [47]

    2026.GPT-5.2 Model Details.https://platform.ope nai.com/docs/models/gpt-5.2Accessed: 2026-01-06

    OpenAI Platform. 2026.GPT-5.2 Model Details.https://platform.ope nai.com/docs/models/gpt-5.2Accessed: 2026-01-06

    2026

  48. [48]

    Chaoyi Ruan, Chao Bi, Kaiwen Zheng, Ziji Shi, Xinyi Wan, and Jialin Li

  49. [49]

    2023.{ServiceRouter}: Hyperscale and minimal cost service mesh at meta

    Harshit Saokar, Soteris Demetriou, Nick Magerko, Max Kontorovich, Josh Kirstein, Margot Leibold, Dimitrios Skarlatos, Hitesh Khandelwal, and Chunqiang Tang. 2023.{ServiceRouter}: Hyperscale and minimal cost service mesh at meta. In17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 969–985

    2023

  50. [50]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789(2023)

    Pith/arXiv arXiv 2023

  51. [51]

    Souhaila Serbout and Cesare Pautasso. 2024. APIstic: A large collection of OpenAPI metrics. InProceedings of the 21st International Conference on Mining Software Repositories. 265–277

    2024

  52. [52]

    2026.Swagger RESTful API Documentation Specification

    Swagger. 2026.Swagger RESTful API Documentation Specification. https://docs.swagger.io/spec.htmlAccessed: 2026-01-06

    2026

  53. [53]

    Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards end- to-end optimization of llm-based applications with ayo. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1302–1316

    2025

  54. [54]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2023), 68539–68551

    2023

  55. [55]

    2026.Qwen3 Max Model Details.https://bailian.console

    Qwen Team. 2026.Qwen3 Max Model Details.https://bailian.console. aliyun.com/?tab=model#/model-market/detail/qwen3-max-2025-09- 23Accessed: 2026-01-06

    2026

  56. [56]

    Chenxu Wang, Xumiao Zhang, Runwei Lu, Xianshang Lin, Xuan Zeng, Xinlei Zhang, Zhe An, Gongwei Wu, Jiaqi Gao, Chen Tian, et al. 2025. Towards llm-based failure localization in production-scale networks. InProceedings of the ACM SIGCOMM 2025 Conference. 496–511

    2025

  57. [57]

    2024.{NetAssistant}: Dialogue based network diagnosis in data center networks

    Haopei Wang, Anubhavnidhi Abhashkumar, Changyu Lin, Tianrong Zhang, Xiaoming Gu, Ning Ma, Chang Wu, Songlin Liu, Wei Zhou, Yongbin Dong, et al. 2024.{NetAssistant}: Dialogue based network diagnosis in data center networks. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 2011–2024

    2024

  58. [58]

    2026.Qwen Plus Model Details.https://bailian.console.al iyun.com/?tab=model#/model-market/detail/qwen-plus-2025-12-01 Accessed: 2026-01-06

    Qwen Team. 2026.Qwen Plus Model Details.https://bailian.console.al iyun.com/?tab=model#/model-market/detail/qwen-plus-2025-12-01 Accessed: 2026-01-06

    2026

  59. [59]

    Shujun Wang, Yongqiang Tian, and Dengcheng He. 2023. Improving API Documentation Comprehensibility via Continuous Optimization and Multilingual SDK.arXiv preprint arXiv:2303.13828(2023)

    Pith/arXiv arXiv 2023

  60. [60]

    Zhaodong Wang, Samuel Lin, Guanqing Yan, Soudeh Ghorbani, Min- lan Yu, Jiawei Zhou, Nathan Hu, Lopa Baruah, Sam Peters, Srikanth Kamath, et al. 2025. Intent-driven network management with multi- agent LLMs: The confucius framework. InProceedings of the ACM SIGCOMM 2025 Conference. 347–362

    2025

  61. [61]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  62. [62]

    Shujun Wang, Yongqiang Tian, and Dengcheng He. 2023. gDoc: Au- tomatic generation of structured API documentation. InCompanion Proceedings of the ACM Web Conference 2023. 53–56

    2023

  63. [63]

    Georg Wölflein, Dyke Ferber, Daniel Truhn, Ognjen Arandjelovic, and Jakob Nikolas Kather. 2025. Llm agents making agent tools. InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 26092–26130

    2025

  64. [64]

    Duo Wu, Xianda Wang, Yaqi Qiao, Zhi Wang, Junchen Jiang, Shuguang Cui, and Fangxin Wang. 2024. Netllm: Adapting large language models for networking. InProceedings of the ACM SIGCOMM 2024 Conference. 661–678

    2024

  65. [65]

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents.arXiv preprint arXiv:2502.12110(2025). 14

    Pith/arXiv arXiv 2025

  66. [66]

    2024.{QDSR}: Accelerating Layer-7 Load Balancing by Direct Server Return with{QUIC}

    Ziqi Wei, Zhiqiang Wang, Qing Li, Yuan Yang, Cheng Luo, Fuyu Wang, Yong Jiang, Sijie Yang, and Zhenhui Yuan. 2024.{QDSR}: Accelerating Layer-7 Load Balancing by Direct Server Return with{QUIC}. In2024 USENIX Annual Technical Conference (USENIX ATC 24). 715–730

    2024

  67. [67]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems36 (2023), 11809–11822

    2023

  68. [68]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations

    2022

  69. [69]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs.Advances in neural information processing systems37 (2024), 62557–62583

    2024

  70. [70]

    Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, et al

  71. [75]

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang

  72. [76]

    InProceedings of the AAAI Conference on Artificial Intelligence, Vol

    Memorybank: Enhancing large language models with long-term memory. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19724–19731. 15 Appendix A Additional Lessons Lesson 5: DNS locality–based VIP selection for multi- AZ Failover.To improve multi-region and multi-AZ re- silience on the gateway -> MCP service data path, we imple- men...

  73. [77]

    Frontend Streamable HTTP, backend Streamable HTTP:the request/response completes on the receiv- ing gateway instance; the gateway consults Session Meta Store as needed to maintain backend session consistency

  74. [78]

    Frontend HTTP+SSE, backend HTTP+SSE:the key requirement is that requests carry Session-ID:Backend so that backend responses are delivered to the correct gateway–backend SSE channel. Any gateway instance may issue the backend request; the backend uses Session- ID:Backend to emit events on the corresponding SSE chan- nel, and the gateway instance holding th...

  75. [79]

    The SSE-owning instance subscribes, executes the request against the back- end using Streamable HTTP, and streams the response back to the client over the established SSE channel

    Frontend HTTP+SSE, backend Streamable HTTP: because the client-facing SSE channel is bound to a spe- cific gateway instance but requests may land elsewhere, the non-owning instance publishes the request (keyed by Session-ID:Frontend) via Pub/Sub. The SSE-owning instance subscribes, executes the request against the back- end using Streamable HTTP, and stre...

  76. [80]

    re- quest accessing instance

    Frontend Streamable HTTP, backend HTTP+SSE:the HTTP response must return from the gateway instance that received the request, but backend responses arrive at the gateway instance owning the gateway–backend SSE channel. The request-receiving instance initiates the back- end call and records a Correlation-ID; the SSE-owning in- stance publishes received bac...

  77. [2023]

    InProceedings of the 29th symposium on operating systems principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles. 611–626

  78. [2024]

    AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework.arXiv preprint arXiv:2412.10422(2024)

    Pith/arXiv arXiv 2024

  79. [2025]

    A survey of ai agent protocols.arXiv preprint arXiv:2504.16736 (2025)

    Pith/arXiv arXiv 2025

  80. [2026]

    In23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26)

    Cortex: Achieving{Low-Latency},{Cost-Efficient} Remote Data Access For{LLM} via{Semantic-Aware} Knowledge Caching. In23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). 2407–2421