Towards a Middleware for Large Language Models

Florian Knauf; Jan S. Rellermeyer; Man Ngo; Narcisa Guran; Stefan Petrescu

REVIEW 4 major objections 5 minor 69 references

Towards a Middleware for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A middleware architecture for self-hosted LLMs that turns the LLM into an enterprise service gateway, with a calculator proof showing the pattern works.

desk verdict A well-scoped vision paper for LLM middleware whose gateway claim is plausible but rests on a calculator demo and deferred scaling; the authors are honest about the gap. read the letter →

arxiv 2411.14513 v1 pith:634RX6TH submitted 2024-11-21 cs.SE cs.CL

Narcisa Guran , Florian Knauf , Man Ngo , Stefan Petrescu , Jan S. Rellermeyer This is my paper · ORCID

classification cs.SEcs.CL

keywords largelanguagemodelsmiddlewareLLMasaServiceGatewaydiscoveryroutingretrieval-augmentedgenerationself-hosting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that self-hosting LLMs in enterprises requires a middleware layer of its own, because an LLM service is more than the model: it carries KV-cache session state, GPU memory limits, retrieval context, and a semantic gap to existing microservices. The proposed architecture covers a baseline LLM-as-a-Service deployment with user registry, scheduler, cache, observability, and explainability components, and a fuller "LLM as a Gateway" mode in which the model itself handles service discovery, parameter extraction, and protocol binding. The paper's proof of concept connects an LLM to a calculator service and reports that passing arithmetic through the service holds accuracy near 100 percent even with twenty arguments, while direct LLM calculation falls to zero percent by five arguments. The broader claim is that with the right middleware, LLMs can absorb parts of the traditional middleware stack and still provide reliable access to enterprise services.

What carries the argument

The load-bearing object is the two-step prompt-binding pattern, used twice: once to turn service discovery into a registry-ranking task, and once to turn parameter binding into a JSON-extraction task. A Service Identifier backed by a vector database retrieves candidate service descriptions filtered by user permissions, and an Execution Graph, modeled after a dataflow graph, chains the selected services, stopping for user input when information is missing and validating each intermediate call with a parser. The calculator experiment is the minimal demonstration that this pattern converts an unreliable arithmetic LLM into a reliable calculator-facing oracle.

What would settle it

Run the paper's two prompt templates over a service registry with dozens to hundreds of services of varied descriptions, measuring end-to-end task accuracy and latency; if accuracy falls below a usable threshold before reaching realistic enterprise registry sizes, the central claim that an LLM can serve as service discovery and binding middleware fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a middleware layer can make self-hosted LLMs practical in enterprises, and that in the most integrated form the LLM becomes the gateway to the application ecosystem: it selects the right service from a registry, turns a natural-language prompt into a concrete operation call with parameters, and translates the result back into language, thereby absorbing service discovery, binding, and protocol adaptation that normally live in middleware. This is supported by a two-step prompting procedure: first the model is shown the registry's application descriptions and asked which app fits the prompt; then it is shown the allowed operations and asked to return a JSON list pairing operations with numbers. In the evaluation, that pattern kept accuracy at 93-100 percent as argument counts grew to 5-20, where the unaided model scored 0-1 percent. The surrounding middleware, consisting of Service Registry, Scheduler, Service Identifier, and Execution Graph, is what the paper claims can provide the guarantees of correctness, permissions, and caching that the model alone lacks.

Load-bearing premise

The architecture depends on the assumption that an LLM given registry metadata keeps identifying services and extracting parameters correctly as the number of services and the complexity of enterprise intents grow; the only demonstration is a calculator with a handful of operations, and the paper defers Execution Graph scaling to future work.

Editorial extensions

If this is right

Self-hosted LLMs can be deployed with middleware-level features such as sticky GPU scheduling, semantic caching, and permission-aware service lookup, so enterprises can avoid cloud dependence for privacy and cost reasons.
Service discovery and binding can be implemented as ranking and extraction over a service registry, using established search-ranking techniques in the small-registry regime and two-tower pre-embedding when the registry grows.
Routing arithmetic-style prompts through an external service turns a model that fails at five arguments into one that stays at 93-100 percent accuracy at five to twenty arguments.
An LLM gateway can chain multiple services in an Execution Graph, pausing mid-chain to request missing information from the user and resuming once it arrives.
Deterministic guarantees for LLM responses come from collaboration with external services rather than from the model alone, which is why the middleware layer matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural test the paper leaves implicit: measure the gateway's accuracy as the service registry grows from one to hundreds of entries; the size at which accuracy drops is the economic boundary between cross-attention routing and two-tower pre-routing.
The same two-step binding pattern could generalize to LLM-to-LLM communication by treating another model's output as a service response, which is the multi-LLM scenario the paper names but does not implement.
The vector database used for the Service Identifier could double as the semantic cache for response caching, so the middleware might need only one embedding index rather than two separate stores.
The gateway pattern redraws the middleware map: if protocol adaptation lives in the model, then service interface evolution may become a prompt-engineering concern rather than an IDL or API migration concern.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this is a vision paper, not a systems paper, and it is a fairly good one. The genuinely new piece is treating the LLM as a gateway that performs service discovery and parameter binding. The authors actually build a small prototype and test it against ground truth, so the idea is not vacuous.

What it does well: the architecture is clearly drawn and the requirements are sensible. The discussion of KV caches, model caching, sticky routing, and observability identifies real deployment concerns for self-hosted LLMs. The related work on LangChain, Bedrock, and Alexa is fair and positions the contribution well. I also credit the authors for writing in the paper itself that their LLM-based binding "provides significantly fewer guarantees compared to traditional middleware" – that is the right qualifier for a vision.

Where the soft spots are: the empirical support is thin. The only experiment is a calculator with 2 to 20 integer arguments, 100 prompts per cell, no error bars, and no model version detail. That is enough to show an LLM can route arithmetic to one tool, but the gateway vision depends on scaling to many services and to multi-step workflows. Section 7.2 explicitly defers the execution graph scaling and the service-registry scaling – exactly the parts that would load-bear the "absorb functionality" claim. So the abstract overclaims relative to the evidence. The paper would be more honest as a research agenda plus existence proof, not a supported architecture. None of this is disqualifying for a vision paper, but it should be presented as such.

Who this is for: people thinking about self-hosted LLM serving and about using LLMs in enterprise microservice ecosystems. It is a good reading-group piece for discussing what counts as evidence. I would not cite it for the evaluation, but I would cite it as a named architectural vision for LLM middleware. A serious referee should engage with it; I would recommend bringing it to a venue that accepts position papers and asking the authors to temper the claims.

Referee Report

4 major / 5 minor

Summary. The paper argues that deploying large language models in enterprises requires a dedicated middleware layer that addresses session state, scheduling, caching, observability, explainability, and integration with existing microservices. It distinguishes a baseline ``LLM as a Service'' architecture from a more ambitious ``LLM as a Gateway'' architecture in which the LLM performs service discovery, parameter binding, and protocol adaptation, potentially absorbing parts of the traditional middleware stack. The paper outlines these components, proposes prompt-based mechanisms for service identification and binding, and evaluates a prototype that connects a Llama 3 model to a calculator service. The evaluation shows that the LLM-plus-calculator combination outperforms the bare LLM on arithmetic prompts, but the scaling of the Execution Graph and Service Identifier to realistic service registries is explicitly left to future work.

Significance. If the vision were validated, the paper would contribute a useful systems-level research agenda for self-hosted LLMs, a clear architecture separating baseline and gateway use cases, and a concrete initial prototype. The paper is strongest in its taxonomy of deployment challenges (Section 3), its separation of ``LLM as a Service'' from ``LLM as a Gateway'' (Sections 4 and 5), and its candid enumeration of open questions in Section 7.2. It does not claim more than an existence proof in the evaluation, and it explicitly identifies the scaling questions that must be answered before the gateway vision is credible. The calculator experiment is not circular because it compares against external ground truth, and the paper does not fit parameters to its own claims. However, the contribution at this stage is a research agenda plus a small proof-of-concept, not a validated middleware system.

major comments (4)

[Section 7.1, Table 1] The only quantitative evidence for the gateway claims is a calculator-arithmetic experiment. Each row of Table 1 reports 100 prompts, but there are no repetitions, no variance measures, no confidence intervals, and the specific Llama 3 variant and inference setup are not stated; at 5 arguments the LLM+Calculator accuracy drops to 93/100, and the error analysis does not separate service-identification failures from parameter-extraction failures. This evidence does not by itself demonstrate that an LLM can route and bind services with the reliability a middleware gateway would require.
[Section 5.2] The service identification and binding mechanism relies on injecting the full registry metadata and allowed operations into the system prompt. The manuscript states in Section 5.2 that this approach provides "significantly fewer guarantees compared to traditional middleware." This is a serious caveat that directly limits the abstract's claim that an LLM can "absorb functionality traditionally attributed to the middleware"; the paper should either present evidence that the reliability gap can be closed (for example, through the validation layer in the Execution Graph) or substantially temper the central claim.
[Section 7.2] Section 7.2 explicitly defers to future work the scaling of the Execution Graph Generator, including the number of services in the registry, the amount of service metadata appended to prompts, and the use of a vector database. Because the Service Identifier and Execution Graph are the components that would turn a single-tool demo into an enterprise gateway, deferring their evaluation leaves the central architectural claim unvalidated; a proof of concept with one calculator service is not sufficient support.
[Section 5.1.2] The proposed scaling path of a two-tower pre-router is incomplete in a load-bearing way: the authors state that "the extension of this method to parameter extraction is less obvious," and parameter extraction is required to bind a discovered service to operations and arguments. Without a concrete mechanism or experimental evidence for the extraction step at scale, the paper does not establish a path from the prototype to a scalable gateway.

minor comments (5)

[Table 1 caption / Section 7.1] The model is referred to as "LLama3" in Table 1 and "LLama3 [40]" in the text; please use the correct name and specify the exact variant (for example, 8B or 70B, bf16, etc.) and the serving stack used for the measurements.
[Section 7.1] The reported response times (for example, 1.031 s for two arguments) are given to three decimal places, but no GPU model, batch size, number of repetitions, or standard deviation is provided; please add these details so the reader can interpret the resource-contention claims.
[Figures 6 and 7] Figures 6 and 7 appear to show the same schematic in the submitted text; if they are meant to illustrate different conversations (an idealized one and a precision-critical one), the figures should be distinct and their captions clarified.
[Section 4.2] The term "sticky-routing policy" is introduced without a definition or citation; please clarify what property is pinned to which session or user.
[Section 6.4 versus Section 7.2] The text refers to both the "Execution Graph" and the "Execution Graph Generator"; please use consistent naming so it is clear whether Section 7.2 is evaluating the generator alone or the whole execution component.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are architectural vision plus an externally validated calculator prototype; no fitted input is relabeled as a prediction and no load-bearing self-citation chain is present.

full rationale

The paper does not derive predictions from fitted parameters. Its central contribution is a proposed middleware architecture and a proof-of-concept evaluation. The Section 7.1 calculator experiment prompts an LLM to route arithmetic prompts to a calculator and measures accuracy against ground-truth expected results; this is an external benchmark, not a quantity reconstructed from the model's own assumptions. The latency and GPU figures in Sections 3.5 and 3.6 are borrowed from cited external sources, which is normal use of prior evidence rather than circularity. The authors self-identify limitations: Section 5.2 states that the LLM-based service binding approach 'provides significantly fewer guarantees compared to traditional middleware,' and Section 7.2 explicitly defers scaling of the Execution Graph Generator, including registry size and metadata volume, to future work. These caveats weaken the strength of the gateway vision but do not constitute circular reasoning. No equations are reused as results, no fitted parameter is renamed as a prediction, and no self-citation is load-bearing. The honest finding is therefore no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 3 assumptions · 3 invented entities

The central vision rests on assumptions about LLM reliability in service discovery and binding, and about the feasibility of describing enterprise services as utterance-based skills. The prototype tests only one tiny case, so these assumptions are largely unvalidated. No numeric free parameters are fitted in the paper.

assumptions (3)

domain assumption LLM prompts containing service registry metadata will identify the correct service and extract parameters with sufficient reliability.
Used in Section 5.2 to replace traditional service discovery and binding. Only tested on a calculator with a handful of operations, so generalization to realistic enterprise service catalogs is assumed.
domain assumption Enterprise services can be described by a set of procedures, parameters, and example utterances, following the Amazon Alexa skill model.
Section 5.1.1 assumes this interface model for the LLM gateway, which restricts applicability to services that can be specified in this utterance-based form.
domain assumption Natural language is an acceptable interface for service discovery in enterprise environments despite its imprecision.
Section 5.1 argues that the semantic gap between natural language and protocols can be bridged by ranking or LLM-based matching, but the reliability of this approach is not established beyond the toy example.

invented entities (3)

LLM-as-Gateway architectural pattern
purpose: To let the LLM act as the front end and integrator for enterprise microservices, absorbing service discovery, binding, and protocol adaptation.
Only demonstrated in a toy calculator scenario; no evaluation of the gateway pattern in a realistic service ecosystem is provided.
Execution Graph component
purpose: To maintain a chain of services that loop until the initial prompt is resolved, with caching and parser validation.
Described in Section 6.4; scalability is explicitly deferred to future work in Section 7.2.
Service Identifier component
purpose: To identify services using a vector database of service descriptions, generating candidate execution graphs.
The prototype uses prompt-based identification directly, not the vector database mechanism; the vector DB path is described but not implemented or measured.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards a Middleware for Large Language Models." pith.science (2026). https://pith.science/paper/634RX6TH

@misc{pith2026241114513,
  author       = {Pith},
  title        = {Pith review of: Towards a Middleware for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/634RX6TH}},
  note         = {Machine review of arXiv:2411.14513}
}

read the original abstract

Large language models have gained widespread popularity for their ability to process natural language inputs and generate insights derived from their training data, nearing the qualities of true artificial intelligence. This advancement has prompted enterprises worldwide to integrate LLMs into their services. So far, this effort is dominated by commercial cloud-based solutions like OpenAI's ChatGPT and Microsoft Azure. As the technology matures, however, there is a strong incentive for independence from major cloud providers through self-hosting "LLM as a Service", driven by privacy, cost, and customization needs. In practice, hosting LLMs independently presents significant challenges due to their complexity and integration issues with existing systems. In this paper, we discuss our vision for a forward-looking middleware system architecture that facilitates the deployment and adoption of LLMs in enterprises, even for advanced use cases in which we foresee LLMs to serve as gateways to a complete application ecosystem and, to some degree, absorb functionality traditionally attributed to the middleware.

Figures

Figures reproduced from arXiv: 2411.14513 by the authors.

**Figure 2.** LLM as a microservice. 3.3 Resource Allocation and Multi-Tenancy Systems like Docker (Swarm) and Kubernetes have originally been designed around managing CPU resources effectively. In contrast, LLMs are typically deployed on GPUs due to the much higher performance caused by the higher degree of hardware parallelism. While strategic loopholes have been introduced into the underlying containers and better matching was… view at source ↗

**Figure 3.** Baseline use-case. The typical scenario for using an LLM is when no external services are considered. In this case, the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: User Registry. To access the framework, every user has an AUTH key and Access Certificate associated. The AUTH key is intended to enable a programmatic way to grant or restrict access to the middleware. The Access Certificate essentially contains all access roles assoc…

**Figure 5.** Figure 5: LLM as a gateway. Natural language is not an ideal format for data transfer between microservices but a great way to communicate with humans. As such, a natural role for LLMs in a service ecosystem is to bridge the gap between human-understandability and machineunde…

**Figure 6.** Figure 6: Idealized conversation with an LLM gateway. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 8.** Figure 8: Cross-attention service router (8a) and two-tower [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 7.** Figure 7: Example conversation highlighting the need for [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 9.** Figure 9: LLM as a Gateway use-case. Proposed middleware for LLM applications: compared to the baseline scenario (Figure 3), [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

**Figure 10.** Figure 10: Scheduler. This component is responsible for assigning workloads to available workers (Request Dispatcher), based on metadata about the service session status and user service permissions (Service Sessions). This way, caching can be used more effectively, while the ex…

**Figure 11.** Figure 11: Example of end-to-end execution for LLMs as a [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

**Figure 12.** Figure 12: LLM + Calculator Application. Zero-shot prompting is used for querying the LLM — we prompt the model without potentially augmenting its capabilities using chainof-thought reasoning or similar strategies. 7.2 Scaling of LLM Execution Graph Generator As the Execution …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 53 canonical work pages

[1]

Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Man- junath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A system for Lar...

work page 2016
[2]

Gpt-4 technical report, 2024

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2024

work page 2024
[3]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

2023
[4]

Define the dialog to collect and confirm required informa- tion

Amazon Alexa. Define the dialog to collect and confirm required informa- tion. https://developer.amazon.com/en-US/docs/alexa/custom-skills/define-the- dialog-to-collect-and-confirm-required-information.html, 2024. Accessed: 2024- 05-30

work page 2024
[5]

Dialog management with alexa conversations

Amazon Alexa. Dialog management with alexa conversations. https://developer. amazon.com/en-US/alexa/alexa-skills-kit/dialog-management, 2024. Accessed: 2024-05-30

work page 2024
[6]

Agents for amazon bedrock

Amazon. Agents for amazon bedrock. https://docs.aws.amazon.com/bedrock/ latest/userguide/agents.html, 2024. Accessed: 2024-04-16

work page 2024
[7]

Meet claude

Anthropic. Meet claude. https://www.anthropic.com/claude, 2024. Accessed: 2024-04-16

work page 2024
[8]

GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings

Fu Bang. GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Liling Tan, Dmitrijs Milajevs, Geeticka Chauhan, Jeremy Gwinnup, and Elijah Rippeth, editors, Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 212–218, Singapore, December 2023. Associ...

work page 2023

Show all 69 references

[9]

Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar

Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval. CoRR, abs/2002.03932, 2020

2002 arXiv
[10]

Punica: Multi-tenant lora serving, 2023

Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krish- namurthy. Punica: Multi-tenant lora serving, 2023

2023
[11]

Frugalgpt: How to use large language models while reducing cost and improving performance, 2023

Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance, 2023

2023
[12]

K. R. Chowdhary. Natural Language Processing, pages 603–649. Springer India, New Delhi, 2020

2020
[13]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vi...

2023
[14]

Llm inference performance engineering: Best practices

Databricks. Llm inference performance engineering: Best practices. https://www.databricks.com/blog/llm-inference-performance-engineering- best-practices, 2024. Accessed: 2024-05-30

2024
[15]

Challenges in service discovery for microservices deployed in a kubernetes cluster–a case study

Baasanjargal Erdenebat, Bayarjargal Bud, and Tamás Kozsik. Challenges in service discovery for microservices deployed in a kubernetes cluster–a case study. Infocommunications Journal, 15(SI):69–75, 2023

2023
[16]

Direct access, High-Performance memory disaggregation with DirectCXL

Donghyun Gouk, Sangwon Lee, Miryeong Kwon, and Myoungsoo Jung. Direct access, High-Performance memory disaggregation with DirectCXL. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) , pages 287–294, Carlsbad, CA, July 2022. USENIX Association

2022
[17]

Service location protocol: Automatic discovery of ip network services

Erik Guttman. Service location protocol: Automatic discovery of ip network services. IEEE Internet computing, 3(4):71–80, 1999

1999
[18]

Learning multiple layers of representation.Trends in cognitive sciences, 11(10):428–434, 2007

Geoffrey E Hinton. Learning multiple layers of representation.Trends in cognitive sciences, 11(10):428–434, 2007

2007
[19]

Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In The 2023 Conference on Empirical Methods in Natural Language Pro...

2023
[20]

A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

2023
[21]

Continuous delivery: reliable software releases through build, test, and deployment automation

Jez Humble and David Farley. Continuous delivery: reliable software releases through build, test, and deployment automation . Pearson Education, 2010

2010
[22]

do you follow me?

Léo Jacqmin, Lina M. Rojas-Barahona, and Benoit Favre. "do you follow me?": A survey of recent approaches in dialogue state tracking, 2022

2022
[23]

Towards mitigating llm hallucination via self reflection

Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating llm hallucination via self reflection. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2023 , pages 1827–1843, 2023

2023
[24]

Colbert: Efficient and effective passage search via contextualized late interaction over bert

Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, pages 39–48, New York...

2020
[25]

Imagenet classification with deep convolutional neural networks

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012

2012
[26]

A survey on out-of-distribution detection in nlp, 2023

Hao Lang, Yinhe Zheng, Yixuan Li, Jian Sun, Fei Huang, and Yongbin Li. A survey on out-of-distribution detection in nlp, 2023

2023
[27]

Conceptual guide

LangChain. Conceptual guide. https://python.langchain.com/v0.2/docs/ concepts/, 2024. Accessed: 2024-05-30

2024
[28]

Langsmith tracing

LangChain. Langsmith tracing. https://docs.smith.langchain.com/concepts/ tracing, 2024. Accessed: 2024-05-28

2024
[29]

Machine learning in bioinformatics

Pedro Larranaga, Borja Calvo, Roberto Santana, Concha Bielza, Josu Galdiano, Inaki Inza, José A Lozano, Rubén Armananzas, Guzmán Santafé, Aritz Pérez, et al. Machine learning in bioinformatics. Briefings in bioinformatics, 7(1):86–112, 2006

2006
[30]

Dialogue state tracking with a language model using schema-driven prompting

Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Dialogue state tracking with a language model using schema-driven prompting. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural...

2021
[31]

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M...

2020
[32]

Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Inf. Proc. Systems , 36, 2024

2024
[33]

Proactive conversational agents in the post-chatgpt world

Lizi Liao, Grace Hui Yang, and Chirag Shah. Proactive conversational agents in the post-chatgpt world. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 3452– 3455, 2023

2023
[34]

A high performance, scalable dns service for very large scale container cloud platforms

Haifeng Liu, Shugang Chen, Yongcheng Bao, Wanli Yang, Yuan Chen, Wei Ding, and Huasong Shan. A high performance, scalable dns service for very large scale container cloud platforms. In Proceedings of the 19th International Middleware Conference Industry, pages 39–45, 2018

2018
[35]

Gonzalez, Ion Stoica, and Matei Zaharia

Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E. Gonzalez, Ion Stoica, and Matei Zaharia. Optimizing llm queries in relational workloads, 2024

2024
[36]

Learn- ing under concept drift: A review

Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. Learn- ing under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346–2363, 2019

2019
[37]

PrivateGPT

Iván Martínez Toro, Daniel Gallego Vico, and Pablo Orgaz. PrivateGPT. https: //github.com/imartinez/privateGPT, May 2023

2023
[38]

In defense of dual-encoders for neural ranking

Aditya Menon, Sadeep Jayasumana, Ankit Singh Rawat, Seungyeon Kim, Sashank Reddi, and Sanjiv Kumar. In defense of dual-encoders for neural ranking. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th ...

2022
[39]

A qualitative study of application-level caching

Jhonny Mertz and Ingrid Nunes. A qualitative study of application-level caching. IEEE Transactions on Software Engineering , 43(9):798–816, 2017. Towards a Middleware for Large Language Models

2017
[40]

Introducing meta llama 3: The most capable openly available llm to date

Meta. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/, 2024. Accessed: 2024-05-20

2024
[41]

Large language models: A survey, 2024

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024

2024
[42]

Agile cold starts for scalable serverless

Anup Mohan, Harshad Sane, Kshitij Doshi, Saikrishna Edupuganti, Naren Nayak, and Vadim Sukhomlinov. Agile cold starts for scalable serverless. In11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19) , Renton, WA, July
[43]

Effi- cient large-scale language model training on gpu clusters using megatron-lm

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Effi- cient large-scale language model training on gpu clusters...

2021
[44]

Applications of large language models (llms) in business analytics–exemplary use cases in data preparation tasks

Mehran Nasseri, Patrick Brandtner, Robert Zimmermann, Taha Falatouri, Farzaneh Darbanian, and Tobechi Obinwanne. Applications of large language models (llms) in business analytics–exemplary use cases in data preparation tasks. In International Conference on Human-Computer Inte...

2023
[45]

Aladdin: Joint placement and scaling for slo-aware llm serving, 2024

Chengyi Nie, Rodrigo Fonseca, and Zhenhua Liu. Aladdin: Joint placement and scaling for slo-aware llm serving, 2024

2024
[46]

Overview of seldon core components ⁄pilcrow

NVidia. Overview of seldon core components ⁄pilcrow. https://www.nvidia.com/de- de/ai-data-science/products/triton-inference-server, 2024. Accessed: 2024-05-29

2024
[47]

Semantics-based automated service discovery

Aabhas V Paliwal, Basit Shafiq, Jaideep Vaidya, Hui Xiong, and Nabil Adam. Semantics-based automated service discovery. IEEE Transactions on Services Computing, 5(2):260–275, 2011

2011
[48]

Efficiently scaling transformer inference

Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Brad- bury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In D. Sond, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems 5 pr...

2023
[49]

Pratheek, Neha Jawalkar, and Arkaprava Basu

B. Pratheek, Neha Jawalkar, and Arkaprava Basu. Improving gpu multi-tenancy with page walk stealing. In 2021 IEEE International Symposium on High- Performance Computer Architecture (HPCA), pages 626–639, 2021

2021
[50]

Overview of seldon core components ⁄pilcrow

Seldon. Overview of seldon core components ⁄pilcrow. https://docs.seldon.io/projects/ seldon-core/en/latest/workflow/overview.html, 2024. Accessed: 2024-05-29

2024
[51]

Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices

Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE access, 5:3909–3943, 2017

2017
[52]

Fast transformer decoding: One write-head is all you need, 2019

Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019

2019
[53]

Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

2020
[54]

Computer vision

George Stockman and Linda G Shapiro. Computer vision. Prentice Hall PTR, 2001

2001
[55]

Gptcache : A library for creating semantic cache for llm queries

Zilliz Tech. Gptcache : A library for creating semantic cache for llm queries. https://github.com/zilliztech/GPTCache, 2024. Accessed: 2024-05-30

2024
[56]

Creating large language model applications utilizing langchain: A primer on developing llm apps fast

Oguzhan Topsakal and Tahir Cetin Akinci. Creating large language model applications utilizing langchain: A primer on developing llm apps fast. In In- ternational Conference on Applied Engineering and Natural Sciences , volume 1, pages 1050–1056, 2023

2023
[57]

Deloitte UK. Open vs. closed-source generative ai. https://www2.deloitte.com/ uk/en/blog/ai-institute/2023/open-vs-closed-source-generative-ai.html, 2024. Accessed: 2024-05-28

2023
[58]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017

2017 arXiv
[59]

Chain-of-thought prompting elicits reasoning in large language models, 2023

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

2023
[60]

Raise a child in large language model: Towards effective and generalizable fine-tuning

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pag...

2021
[61]

Hallucination is inevitable: An innate limitation of large language models

Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024

2024 arXiv
[62]

React: Synergizing reasoning and acting in language models, 2023

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

2023
[63]

Deltazip: Multi-tenant language model serving via delta compression, 2023

Xiaozhe Yao and Ana Klimovic. Deltazip: Multi-tenant language model serving via delta compression, 2023

2023
[64]

A survey on large language model (llm) security and privacy: The good, the bad, and the ugly

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211, 2024

2024
[65]

Nomad-attention: Efficient llm inference on cpus through multiply- add-free attention, 2024

Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu, and Anshumali Shrivastava. Nomad-attention: Efficient llm inference on cpus through multiply- add-free attention, 2024

2024
[66]

Explainability for large language models: A survey, 2023

Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey, 2023

2023
[67]

Service discovery in pervasive computing environments

Feng Zhu, Matt W Mutka, and Lionel M Ni. Service discovery in pervasive computing environments. IEEE Pervasive computing, 4(4):81–90, 2005

2005
[68]

Large language models for information retrieval: A survey

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107, 2023

2023
[69]

Large language models are built-in autoregressive search engines

Noah Ziems, Wenhao Yu, Zhihan Zhang, and Meng Jiang. Large language models are built-in autoregressive search engines. arXiv preprint arXiv:2305.09612, 2023

2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Man- junath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A system for Lar...

work page 2016

[2] [2]

Gpt-4 technical report, 2024

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2024

work page 2024

[3] [3]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

2023

[4] [4]

Define the dialog to collect and confirm required informa- tion

Amazon Alexa. Define the dialog to collect and confirm required informa- tion. https://developer.amazon.com/en-US/docs/alexa/custom-skills/define-the- dialog-to-collect-and-confirm-required-information.html, 2024. Accessed: 2024- 05-30

work page 2024

[5] [5]

Dialog management with alexa conversations

Amazon Alexa. Dialog management with alexa conversations. https://developer. amazon.com/en-US/alexa/alexa-skills-kit/dialog-management, 2024. Accessed: 2024-05-30

work page 2024

[6] [6]

Agents for amazon bedrock

Amazon. Agents for amazon bedrock. https://docs.aws.amazon.com/bedrock/ latest/userguide/agents.html, 2024. Accessed: 2024-04-16

work page 2024

[7] [7]

Meet claude

Anthropic. Meet claude. https://www.anthropic.com/claude, 2024. Accessed: 2024-04-16

work page 2024

[8] [8]

GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings

Fu Bang. GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Liling Tan, Dmitrijs Milajevs, Geeticka Chauhan, Jeremy Gwinnup, and Elijah Rippeth, editors, Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 212–218, Singapore, December 2023. Associ...

work page 2023