{"id":"6b9b4192-ccd3-4d21-9b32-993e20bedb7e","arxiv_id":"2411.14513","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This paper outlines a middleware architecture for self-hosted LLMs and shows with a calculator example that an LLM can perform service discovery and protocol adaptation.","lead":"This paper proposes a middleware architecture for companies that want to self-host large language models instead of relying on cloud APIs. It also describes a small prototype where an LLM routes requests to a calculator service, and argues that LLMs could eventually absorb parts of traditional middleware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gateway claim rests on unvalidated scaling: only arithmetic routing is tested, and Section 7.2 defers exactly the service-registry scaling on which the vision depends.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the viability of the architecture depends on the LLM reliably identifying services and extracting parameters as the number of services and the complexity of intents grow, while Section 7.1 tests only a calculator and Section 7.2 defers exactly the relevant scaling study. I found additional internal evidence supporting this concern: Section 5.1.2 explicitly leaves parameter extraction via two-tower pre-routing unresolved, and Section 5.2 acknowledges that the LLM-based approach provides fewer guarantees than traditional middleware. These are not contradictions that make the paper unsound, but they do mean the central gateway claim is currently a proof-of-concept rather than a validated architecture. The paper is a clearly written vision with a sensible research agenda, and the calculator experiment is a useful existence proof for LLM-based tool routing. Because the reader already assigned CONDITIONAL with high confidence and correctly identified the scalability assumption as the weak point, my stress-test does not change the verdict. No ad hominem or manufactured concern is warranted; the issue is a straightforward gap between the breadth of the claim and the narrowness of the evaluation.","tokens_in":19076,"tokens_out":3055,"duration_ms":30170,"concrete_test":"Run the Section 5.2 prompt templates against a synthetic service registry with N services (N = 1, 10, 100) using semantically overlapping service descriptions and realistic enterprise prompts (e.g., booking, reporting, data lookup, parameterized queries with dates, strings, and enums). Measure top-1 service identification accuracy, exact parameter extraction accuracy, and valid-JSON response rate over at least 500 prompts per condition. If accuracy or valid-JSON rate declines materially with N, or if non-integer parameter types fail, the LLM-as-gateway claim lacks scaling and generality support. Optionally, implement the two-tower pre-router from Section 5.1.2 and compare end-to-end accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an LLM can serve as a reliable gateway to an enterprise service ecosystem, discovering and binding services with middleware-level guarantees. That claim requires the LLM-based service identifier and parameter extractor to remain accurate as the number and variety of services grow, and it requires the Execution Graph to compose multi-service workflows without accumulating errors. The paper's only evidence is Section 7.1's calculator experiment: one service, simple integer operations, 100 prompts per cell, and no error bars. Section 7.2 explicitly defers scaling of the Execution Graph Generator to future work, including the number of services in the registry and the amount of metadata appended to prompts. Section 5.1.2 proposes a two-tower pre-router as the scaling path, but the authors concede that 'the extension of this method to parameter extraction is less obvious'; parameter extraction is the other half of service routing. The implemented prototype uses the Section 5.2 prompt templates, which dump registry metadata into the prompt; this cannot be assumed to stay accurate or affordable as registries reach enterprise scale. The paper itself states in Section 5.2 that this approach provides 'significantly fewer guarantees compared to traditional middleware,' which undercuts the abstract's vision of absorbing middleware functionality. The evaluation is therefore an existence proof that one small LLM can route arithmetic to one tool, not a demonstration that the proposed middleware can reliably integrate and route between enterprise services.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that deploying large language models in enterprises requires a dedicated middleware layer that addresses session state, scheduling, caching, observability, explainability, and integration with existing microservices. It distinguishes a baseline ``LLM as a Service'' architecture from a more ambitious ``LLM as a Gateway'' architecture in which the LLM performs service discovery, parameter binding, and protocol adaptation, potentially absorbing parts of the traditional middleware stack. The paper outlines these components, proposes prompt-based mechanisms for service identification and binding, and evaluates a prototype that connects a Llama 3 model to a calculator service. The evaluation shows that the LLM-plus-calculator combination outperforms the bare LLM on arithmetic prompts, but the scaling of the Execution Graph and Service Identifier to realistic service registries is explicitly left to future work.","tokens_in":19306,"tokens_out":5917,"duration_ms":53624,"significance":"If the vision were validated, the paper would contribute a useful systems-level research agenda for self-hosted LLMs, a clear architecture separating baseline and gateway use cases, and a concrete initial prototype. The paper is strongest in its taxonomy of deployment challenges (Section 3), its separation of ``LLM as a Service'' from ``LLM as a Gateway'' (Sections 4 and 5), and its candid enumeration of open questions in Section 7.2. It does not claim more than an existence proof in the evaluation, and it explicitly identifies the scaling questions that must be answered before the gateway vision is credible. The calculator experiment is not circular because it compares against external ground truth, and the paper does not fit parameters to its own claims. However, the contribution at this stage is a research agenda plus a small proof-of-concept, not a validated middleware system.","major_comments":[{"comment":"The only quantitative evidence for the gateway claims is a calculator-arithmetic experiment. Each row of Table 1 reports 100 prompts, but there are no repetitions, no variance measures, no confidence intervals, and the specific Llama 3 variant and inference setup are not stated; at 5 arguments the LLM+Calculator accuracy drops to 93/100, and the error analysis does not separate service-identification failures from parameter-extraction failures. This evidence does not by itself demonstrate that an LLM can route and bind services with the reliability a middleware gateway would require.","section":"Section 7.1, Table 1"},{"comment":"The service identification and binding mechanism relies on injecting the full registry metadata and allowed operations into the system prompt. The manuscript states in Section 5.2 that this approach provides \"significantly fewer guarantees compared to traditional middleware.\" This is a serious caveat that directly limits the abstract's claim that an LLM can \"absorb functionality traditionally attributed to the middleware\"; the paper should either present evidence that the reliability gap can be closed (for example, through the validation layer in the Execution Graph) or substantially temper the central claim.","section":"Section 5.2"},{"comment":"Section 7.2 explicitly defers to future work the scaling of the Execution Graph Generator, including the number of services in the registry, the amount of service metadata appended to prompts, and the use of a vector database. Because the Service Identifier and Execution Graph are the components that would turn a single-tool demo into an enterprise gateway, deferring their evaluation leaves the central architectural claim unvalidated; a proof of concept with one calculator service is not sufficient support.","section":"Section 7.2"},{"comment":"The proposed scaling path of a two-tower pre-router is incomplete in a load-bearing way: the authors state that \"the extension of this method to parameter extraction is less obvious,\" and parameter extraction is required to bind a discovered service to operations and arguments. Without a concrete mechanism or experimental evidence for the extraction step at scale, the paper does not establish a path from the prototype to a scalable gateway.","section":"Section 5.1.2"}],"minor_comments":[{"comment":"The model is referred to as \"LLama3\" in Table 1 and \"LLama3 [40]\" in the text; please use the correct name and specify the exact variant (for example, 8B or 70B, bf16, etc.) and the serving stack used for the measurements.","section":"Table 1 caption / Section 7.1"},{"comment":"The reported response times (for example, 1.031 s for two arguments) are given to three decimal places, but no GPU model, batch size, number of repetitions, or standard deviation is provided; please add these details so the reader can interpret the resource-contention claims.","section":"Section 7.1"},{"comment":"Figures 6 and 7 appear to show the same schematic in the submitted text; if they are meant to illustrate different conversations (an idealized one and a precision-critical one), the figures should be distinct and their captions clarified.","section":"Figures 6 and 7"},{"comment":"The term \"sticky-routing policy\" is introduced without a definition or citation; please clarify what property is pinned to which session or user.","section":"Section 4.2"},{"comment":"The text refers to both the \"Execution Graph\" and the \"Execution Graph Generator\"; please use consistent naming so it is clear whether Section 7.2 is evaluating the generator alone or the whole execution component.","section":"Section 6.4 versus Section 7.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best framed as a vision and research-agenda paper. If the venue's bar for systems papers requires end-to-end evaluation, the current evidence is thin; however, the architectural discussion is suitable for a workshop or a vision track. The authors should be asked to align the abstract's strong gateway claims with the caveats in Section 5.2 and to add at least a small multi-service routing experiment or a clearly scoped threat analysis of the scaling assumptions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a vision paper, not a systems paper, and it is a fairly good one. The genuinely new piece is treating the LLM as a gateway that performs service discovery and parameter binding. The authors actually build a small prototype and test it against ground truth, so the idea is not vacuous.\n\nWhat it does well: the architecture is clearly drawn and the requirements are sensible. The discussion of KV caches, model caching, sticky routing, and observability identifies real deployment concerns for self-hosted LLMs. The related work on LangChain, Bedrock, and Alexa is fair and positions the contribution well. I also credit the authors for writing in the paper itself that their LLM-based binding \"provides significantly fewer guarantees compared to traditional middleware\" – that is the right qualifier for a vision.\n\nWhere the soft spots are: the empirical support is thin. The only experiment is a calculator with 2 to 20 integer arguments, 100 prompts per cell, no error bars, and no model version detail. That is enough to show an LLM can route arithmetic to one tool, but the gateway vision depends on scaling to many services and to multi-step workflows. Section 7.2 explicitly defers the execution graph scaling and the service-registry scaling – exactly the parts that would load-bear the \"absorb functionality\" claim. So the abstract overclaims relative to the evidence. The paper would be more honest as a research agenda plus existence proof, not a supported architecture. None of this is disqualifying for a vision paper, but it should be presented as such.\n\nWho this is for: people thinking about self-hosted LLM serving and about using LLMs in enterprise microservice ecosystems. It is a good reading-group piece for discussing what counts as evidence. I would not cite it for the evaluation, but I would cite it as a named architectural vision for LLM middleware. A serious referee should engage with it; I would recommend bringing it to a venue that accepts position papers and asking the authors to temper the claims.","headline":"A well-scoped vision paper for LLM middleware whose gateway claim is plausible but rests on a calculator demo and deferred scaling; the authors are honest about the gap.","tokens_in":19857,"tokens_out":2199,"would_cite":true,"duration_ms":20289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A middleware architecture for self-hosted LLMs that turns the LLM into an enterprise service gateway, with a calculator proof showing the pattern works.","keywords":["large language models","middleware","LLM as a Service","LLM as a Gateway","service discovery","service routing","retrieval-augmented generation","self-hosting"],"falsifier":"Run the paper's two prompt templates over a service registry with dozens to hundreds of services of varied descriptions, measuring end-to-end task accuracy and latency; if accuracy falls below a usable threshold before reaching realistic enterprise registry sizes, the central claim that an LLM can serve as service discovery and binding middleware fails.","tokens_in":18878,"feed_emoji":"🤖","tokens_out":6343,"duration_ms":57621,"temperature":0.7,"pith_summary":"The paper argues that self-hosting LLMs in enterprises requires a middleware layer of its own, because an LLM service is more than the model: it carries KV-cache session state, GPU memory limits, retrieval context, and a semantic gap to existing microservices. The proposed architecture covers a baseline LLM-as-a-Service deployment with user registry, scheduler, cache, observability, and explainability components, and a fuller \"LLM as a Gateway\" mode in which the model itself handles service discovery, parameter extraction, and protocol binding. The paper's proof of concept connects an LLM to a calculator service and reports that passing arithmetic through the service holds accuracy near 100 percent even with twenty arguments, while direct LLM calculation falls to zero percent by five arguments. The broader claim is that with the right middleware, LLMs can absorb parts of the traditional middleware stack and still provide reliable access to enterprise services.","feed_headline":"Middleware blueprint makes LLMs the enterprise gateway","feed_subtitle":"Self-hosted LLMs could handle service discovery and routing; a calculator test shows near-perfect accuracy at 20 arguments.","key_machinery":"The load-bearing object is the two-step prompt-binding pattern, used twice: once to turn service discovery into a registry-ranking task, and once to turn parameter binding into a JSON-extraction task. A Service Identifier backed by a vector database retrieves candidate service descriptions filtered by user permissions, and an Execution Graph, modeled after a dataflow graph, chains the selected services, stopping for user input when information is missing and validating each intermediate call with a parser. The calculator experiment is the minimal demonstration that this pattern converts an unreliable arithmetic LLM into a reliable calculator-facing oracle.","core_discovery":"The paper's central claim is that a middleware layer can make self-hosted LLMs practical in enterprises, and that in the most integrated form the LLM becomes the gateway to the application ecosystem: it selects the right service from a registry, turns a natural-language prompt into a concrete operation call with parameters, and translates the result back into language, thereby absorbing service discovery, binding, and protocol adaptation that normally live in middleware. This is supported by a two-step prompting procedure: first the model is shown the registry's application descriptions and asked which app fits the prompt; then it is shown the allowed operations and asked to return a JSON list pairing operations with numbers. In the evaluation, that pattern kept accuracy at 93-100 percent as argument counts grew to 5-20, where the unaided model scored 0-1 percent. The surrounding middleware, consisting of Service Registry, Scheduler, Service Identifier, and Execution Graph, is what the paper claims can provide the guarantees of correctness, permissions, and caching that the model alone lacks.","pith_inferences":["A natural test the paper leaves implicit: measure the gateway's accuracy as the service registry grows from one to hundreds of entries; the size at which accuracy drops is the economic boundary between cross-attention routing and two-tower pre-routing.","The same two-step binding pattern could generalize to LLM-to-LLM communication by treating another model's output as a service response, which is the multi-LLM scenario the paper names but does not implement.","The vector database used for the Service Identifier could double as the semantic cache for response caching, so the middleware might need only one embedding index rather than two separate stores.","The gateway pattern redraws the middleware map: if protocol adaptation lives in the model, then service interface evolution may become a prompt-engineering concern rather than an IDL or API migration concern."],"forward_implications":["Self-hosted LLMs can be deployed with middleware-level features such as sticky GPU scheduling, semantic caching, and permission-aware service lookup, so enterprises can avoid cloud dependence for privacy and cost reasons.","Service discovery and binding can be implemented as ranking and extraction over a service registry, using established search-ranking techniques in the small-registry regime and two-tower pre-embedding when the registry grows.","Routing arithmetic-style prompts through an external service turns a model that fails at five arguments into one that stays at 93-100 percent accuracy at five to twenty arguments.","An LLM gateway can chain multiple services in an Execution Graph, pausing mid-chain to request missing information from the user and resuming once it arrives.","Deterministic guarantees for LLM responses come from collaboration with external services rather than from the model alone, which is why the middleware layer matters."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the context-grounding mechanism the paper's deployment model builds on.","marker":"[31]"},{"why":"Supplies the open model family used in the calculator proof-of-concept and in the GPU memory sizing example.","marker":"[40]"},{"why":"Supplies the cross-attention re-ranking technique that service routing as utterance ranking adapts.","marker":"[24]"},{"why":"Supplies the two-tower embedding model for the scalable pre-routing variant.","marker":"[9]"},{"why":"Supports the accuracy trade-off that justifies choosing cross-attention routing while the registry is small.","marker":"[38]"},{"why":"ReAct prompting is the mechanism the explainability component uses to interleave reasoning with external actions.","marker":"[62]"},{"why":"The dataflow graph is the stated inspiration for the Execution Graph component.","marker":"[1]"},{"why":"Delta compression is the proposed mechanism for switching between fine-tuned model versions in scheduling and caching.","marker":"[63]"},{"why":"Defines the skill and intent-slot interface that the gateway's service procedure abstraction is modeled after.","marker":"[4]"}],"fun_headline_variants":["LLMs as middleware gateways: 93-100% accuracy on 20 args","Two-step prompt turns LLMs into near-perfect middleware routing","Self-hosted LLMs: middleware vision with near-perfect service calls","LLM gateway via middleware: registry-driven accuracy 93-100%","Middleware blueprint: LLMs as enterprise app gateways"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The architecture depends on the assumption that an LLM given registry metadata keeps identifying services and extracting parameters correctly as the number of services and the complexity of enterprise intents grow; the only demonstration is a calculator with a handful of operations, and the paper defers Execution Graph scaling to future work.","fun_headline_variants_meta":{"raw":{"variants":["LLMs as middleware gateways: 93-100% accuracy on 20 args","Two-step prompt turns LLMs into near-perfect middleware routing","Self-hosted LLMs: middleware vision with near-perfect service calls","LLM gateway via middleware: registry-driven accuracy 93-100%","Middleware blueprint: LLMs as enterprise app gateways"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2082,"prompt_tokens":903,"completion_tokens":1179,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1086}},"tokens_in":519,"tokens_out":1179,"duration_ms":10799,"temperature":1.0,"reasoning_tokens":1086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:30:10.134720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's two prompt templates over a service registry with dozens to hundreds of services of varied descriptions, measuring end-to-end task accuracy and latency; if accuracy falls below a usable threshold before reaching realistic enterprise registry sizes, the central claim that an LLM can serve as service discovery and binding middleware fails.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the context-grounding mechanism the paper's deployment model builds on."},{"cited_title":"Introducing meta llama 3: The most capable openly available llm to date","cited_arxiv_id":null,"evidence_quote":"Supplies the open model family used in the calculator proof-of-concept and in the GPU memory sizing example."},{"cited_title":"Colbert: Efficient and effective passage search via contextualized late interaction over bert","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-attention re-ranking technique that service routing as utterance ranking adapts."},{"cited_title":"In defense of dual-encoders for neural ranking","cited_arxiv_id":null,"evidence_quote":"Supports the accuracy trade-off that justifies choosing cross-attention routing while the registry is small."},{"cited_title":"Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng","cited_arxiv_id":null,"evidence_quote":"The dataflow graph is the stated inspiration for the Execution Graph component."},{"cited_title":"Deltazip: Multi-tenant language model serving via delta compression, 2023","cited_arxiv_id":null,"evidence_quote":"Delta compression is the proposed mechanism for switching between fine-tuned model versions in scheduling and caching."},{"cited_title":"Define the dialog to collect and confirm required informa- tion","cited_arxiv_id":null,"evidence_quote":"Defines the skill and intent-slot interface that the gateway's service procedure abstraction is modeled after."}],"review_version":1}