HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving

Vibha Murthy; Wanlu Li; Xiaoyi Fang; Yufei Ding; Yuke Wang; Zaifeng Pan; Zhengding Hu

REVIEW 4 major objections 6 minor 1 cited by

HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims a runtime that treats heterogeneous RAG workflows as graphs and rewrites them on the fly can lift throughput 1.5x to 5x and cut latency up to 18.2x.

desk verdict A credible RAG-serving system with a real graph abstraction, but the 1.5x–5x gains are not cleanly attributed because no naive async baseline is measured. read the letter →

arxiv 2507.09138 v1 pith:4F6CW2T5 submitted 2025-07-12 cs.DB cs.LG

Zhengding Hu , Vibha Murthy , Zaifeng Pan , Wanlu Li , Xiaoyi Fang , Yufei Ding , Yuke Wang This is my paper

classification cs.DBcs.LG

keywords retrieval-augmentedgenerationRAGservingheterogeneousworkflowsgraph-basedabstractionspeculativeexecutionGPUindexcachingvectorsearchCPU-GPUpipelining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern retrieval-augmented generation requests no longer look alike: some do a single retrieve-then-generate pass, while others decompose questions, generate hypothetical documents, iterate through several retrieval–generation rounds, or compress retrieved passages, and stage lengths vary widely. HedraRAG argues that this heterogeneity should be handled by a runtime that treats every request workflow as a graph and rewrites the graph as it executes, instead of composing fixed sequential stages. On a CPU–GPU server, the system splits stages into fine-grained sub-stages for smooth pipelining, exploits the fact that successive retrieval queries and partial generations stay close to their final embeddings to overlap dependent stages speculatively, and keeps the most frequently searched index clusters resident on the GPU. Across five workflow shapes and three open-domain QA datasets, the paper reports throughput gains over 1.5x and up to 5x compared with existing frameworks, with end-to-end latency reduced by 2.2x to 18.2x at matched request rates.

What carries the argument

RAGraph is the central object: a directed graph whose nodes are generation stages and retrieval stages, with edges carrying data and control flow, including conditional branches. HedraRAG's scheduler repeatedly selects a wavefront of nodes that can run in parallel and applies graph transformations to them—splitting a node into cheaper sub-nodes, reordering cluster searches so that promising clusters are visited first, and inserting speculative edges that let a downstream generation start on partial retrieval results or a retrieval start on partial generation embeddings. These transformations are what make three heterogeneous optimizations (pipelining, semantic speculation, and GPU cluster caching) expressible in one mechanism, and the time-budget formula for sub-stage size is the load-bearing tuning parameter: it trades shorter waits for retrieval against the scheduling overhead, choosing the split that maximizes expected latency improvement.

What would settle it

Feed the system a request stream in which successive retrieval rounds are engineered to be semantically unrelated, so query embeddings between rounds are no closer than random pairs; if the throughput gain over a strictly sequential pipeline then falls below the claimed 1.5x, the intra-request similarity assumption is load-bearing. A second test is to use a corpus whose inverted-file clusters are accessed nearly uniformly, so the GPU cache hit rate drops; if the GPU-indexing speedup disappears, inter-request skewness is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in serving heterogeneous RAG is not retrieval or generation in isolation but their lack of coordination, and that a graph-based runtime can recover the lost efficiency by restructuring each workflow while it runs. The paper introduces RAGraph, in which generation and retrieval are node types and dependencies are edges, and defines four transformation operations—node splitting, reordering, edge addition, and dependency rewiring—that turn coarse sequential stages into fine-grained, overlappable sub-stages. These transformations are applied to wavefronts of pending sub-nodes assembled from concurrent requests and are dispatched to a hybrid CPU–GPU pipeline: generation runs in step-wise batches, retrieval searches run over one or more index clusters, and a scheduler chooses how much to speculate based on measured system utilization and semantic drift. The claimed result is that pipelining, locality-based speculation, and partial GPU indexing together deliver consistent speedups that grow with workflow complexity and with retrieval cost, reaching over 1.5x and up to 5x in throughput.

Load-bearing premise

The load-bearing premise is empirical: real RAG workloads must show locality of two kinds—within a request, consecutive retrieval queries and partial generations stay close to their eventual embeddings, and across requests, a small set of index clusters carries most of the search traffic—because if either regularity fails, speculation rolls back and the GPU cache misses, shrinking the reported gains.

Editorial extensions

If this is right

If the speedups hold, RAG serving should be built as a coordinated runtime over workflow graphs, not as two glued-together components, since the gains come from restructuring the workflow at runtime.
Multi-round workflows and workloads with heavier retrieval, measured by more clusters probed per search, benefit most because they create more chances to pipeline and speculate.
Existing and future optimizations, such as retrieval-aware speculation and index prefetching, can be plugged into the same graph machinery as new edges or nodes without redesigning the scheduler.
With small- and medium-size LLMs, where retrieval is a large fraction of end-to-end time, coordinated retrieval is where the benefit concentrates; with very large models and small databases the bottleneck shifts to generation and the coordination gains diminish.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves open whether the wavefront-transformation machinery transfers to other heterogeneous pairings, such as tool calls or database operators followed by LLM steps; the abstraction suggests it would whenever one stage's output is a vector that predicts the next stage's input.
A testable extension not evaluated in the paper is an adaptive early-commit policy for any long pipeline: stop waiting for a full stage once embedding drift from the final result falls below a threshold, generalizing the 22–50% token-ratio observation.
The GPU-cache versus KV-cache split is chosen offline from throughput curves; an online controller that adapts the split to the observed request mix is a plausible follow-up the paper leaves implicit.
The locality measurements are demonstrated on inverted-file indexes over QA corpora; whether the same speculation and caching benefits appear on other approximate indexes, such as graph-based or quantized indexes, is not tested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a serious systems paper with a genuinely new abstraction (RAGraph) and an extensive evaluation, but the headline speedup attribution is under-supported because no naive async baseline is measured. The 1.5x–5x numbers are plausible, not obviously fabricated, and the internals hang together.

What's actually new: representing RAG workflows as a graph and applying dynamic transformations (node splitting, reordering, speculative edges, dependency rewiring) to coordinate LLM generation and vector retrieval at runtime. That's a reasonable step beyond stage-centric schedulers like Ayo and Cognify, and beyond the caching/speculation work of RAGCache, RaLMSpec, and PipeRAG. The implementation on vLLM and Faiss is real, and the evaluation covers five workflow types, three datasets, three model sizes, and multiple nprobe settings. The microbenchmarks in Figures 16–18 give the individual techniques independent support.

Soft spots, in order of severity. First, the evaluation baseline problem is real: LangChain and FlashRAG are sequential pipelines, so the end-to-end speedup could be dominated by basic CPU-GPU overlap rather than the graph transformations. The paper explicitly shows a naive async scheme has delays in Figure 5(b) but never measures it in Section 6. A single naive-async baseline would settle the attribution. Second, there are no artifacts, no error bars, and no repetition counts; the speculative baselines are reimplemented inside HedraRAG, which makes them hard to trust. Third, the locality assumptions that drive speculation and GPU caching come from three QA datasets; real workloads with weaker inter-query similarity or flatter index access could shrink the gains considerably. None of these is fatal, but together they mean the reported gains are upper-bound-ish.

I would not desk-reject this. It deserves a serious referee. A major revision that adds a naive async baseline, ships code and data, and reports variance could turn the conditional verdict into a solid accept. For a reader in the RAG-serving or LLM-infra space, this is a useful read.

Referee Report

4 major / 6 minor

Summary. The paper presents HedraRAG, a runtime system for serving heterogeneous RAG workflows on hybrid CPU-GPU platforms. Its central idea is RAGraph, a graph-based abstraction through which a scheduler applies transformations—node splitting, reordering, edge addition, and dependency rewiring—to expose and exploit cross-stage parallelism, intra-request semantic similarity, and inter-request index-access skewness. These transformations are realized through fine-grained sub-stage partitioning with dynamic batching, similarity-aware reordering with speculative generation/retrieval, and partial GPU index caching with asynchronous updates. The evaluation compares HedraRAG with LangChain and FlashRAG for one-shot, HyDE, RECOMP, Multistep, and IRG workflows, and with reimplemented RaLMSpec and RAGCache speculative baselines, reporting 1.5x–5x throughput speedups and larger latency reductions across multiple LLMs and nprobe settings.

Significance. If the reported gains hold, HedraRAG addresses a genuine and increasingly important problem: existing RAG serving frameworks treat generation and retrieval as isolated stages, leaving substantial CPU-GPU utilization on the table. The RAGraph abstraction is a genuinely useful way to unify workflow heterogeneity, and the breadth of workflows, models, and index settings considered is a strength. The paper would be a solid systems contribution. However, the central claim is currently under-verified: the evaluation lacks a simple asynchronous-pipeline baseline, the prior speculative baselines are reimplemented inside HedraRAG rather than run as standalone systems, the measurements have no statistical support, and no end-to-end quality metric is reported. These gaps are fixable in scope and do not point to a fundamental flaw in the design, but they need to be addressed before the paper's attribution and headline numbers can be accepted.

major comments (4)

[§3.1, §6.1–6.2] The central attribution of the reported speedups to HedraRAG's coordinated graph transformations is not established, because no asynchronous CPU-GPU overlap baseline is measured. The paper itself acknowledges in Figure 5(b) that naive async integration can exhibit scheduling delays, yet the only overall baselines in Section 6.1 are LangChain and FlashRAG, which Section 2.3 describes as dispatching stages sequentially. Thus the 1.5x–5x gains in Figures 12–14 could be dominated by elementary vLLM-step/Faiss-search overlap rather than by sub-stage partitioning, speculation, or GPU caching. Please add a baseline that runs vLLM and Faiss in separate processes with simple asynchronous pipelining, without the three proposed techniques, and report whether it already captures most of the speedup.
[§6.1, Figure 17] The comparison against RaLMSpec and RAGCache is not a fair evaluation of those prior methods. The text states that "as neither provides open-source access, we enable support for both in HedraRAG by adding speculative execution edges," meaning both are reimplemented inside HedraRAG's own scheduler and measured through its execution substrate. This cannot support the claim in Section 6.3 that HedraRAG achieves 1.06x–1.62x latency speedup over prior methods. Either run the original standalone implementations or provide an ablation that isolates the speculation policy from the surrounding system so the reader can see what is actually being compared.
[§6.2–6.3, Figures 12–18] All latency, throughput, speedup, and speculation-accuracy numbers are single-point measurements with no error bars, no repetition counts, and no confidence intervals, and no code or artifact is provided. System measurements on a shared CPU-GPU server are noisy, and the headline numbers (1.5x–5x throughput gains, up to 18.2x latency reduction) are therefore not quantitatively established. Please report the number of runs and variance for each configuration, and release the artifact or a detailed reproducibility appendix.
[§6.3, Figure 17] The paper claims in Section 3.2 that the optimizations improve latency "without sacrificing result quality," but no end-to-end quality metric is reported for any workflow or configuration. The speculation accuracy defined in Section 6.3 only measures whether partial retrieval results match final retrieval results; it does not measure whether generated answers remain correct after reordering, speculative retrieval, early termination, or rollback. Please report standard QA metrics (e.g., exact match or F1) for the three datasets across all workflows and index settings, with speculation enabled and disabled.

minor comments (6)

[§4.3, §5] Typos and grammatical errors: "senarios" in Section 4.3, "patern" in Section 5, and "the intra-request the semantic similarity" in Section 4.3.
[§6.2, Figure 13] The text reports "speedups of 3.5x and 1.3x" over LangChain and FlashRAG, but the figure shows normalized total time with values above 2 truncated; the relationship between the axis and the reported speedups is confusing and should be clarified.
[References] References [33], [34], and [35] are the same paper (Johnson et al., Billion-scale similarity search with GPUs) and should be collapsed into a single citation.
[§4.2–4.4] Key runtime parameters—mb, beta, tau, gc, cache-update interval, gs, local cache top-k, and delta_s—are introduced but their values are not reported, and no sensitivity study is provided; at minimum, the chosen operating points should be listed and varied.
[Listing 1] The code example uses add_edge(2, lambda s: 1 if s.get("subquestion") else END), where the second argument appears to be a Python lambda rather than a node; clarify the intended API semantics.
[§3.2, Figure 7(b)] The claim that 22–50% of generated tokens put the embedding in the top-1 similarity range lacks methodological detail (which embedding model, how the partial-generation embeddings are computed, and over how many queries); a brief description of the measurement procedure would improve interpretability.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; central speedup claims are measured end-to-end against external baselines, with only tangential non-load-bearing self-citations.

full rationale

HedraRAG's performance claims are not reduced by construction to its inputs. The abstract's 1.5x-5x speedups come from Section 6.2 latency/throughput comparisons against LangChain and FlashRAG, and Section 6.3 compares speculation against RaLMSpec/RAGCache; these are externally benchmarked measurements, not predictions from fitted parameters. The two tuning formulas, Eq. (1) for the time budget m_b and Eq. (2) for GPU memory split, are engineering performance-model choices with stated assumptions (even retrieval arrival; measured t_Retrieval; offline-characterized throughput curves) rather than derivations that define the reported result. The locality observations in Figures 7-9 and the skewness claim in Figure 8 are empirical measurements on open QA datasets; the speculative and caching mechanisms are then evaluated by measured speculation accuracy, cache hit rate, and end-to-end latency, so the techniques are not assumed to be effective by definition. The only author self-citations are to [12,13] on triangle-inequality pruning, mentioned as background in Sections 2.2 and 3.2; the paper explicitly argues those methods are less effective in high-dimensional embedding spaces, so those citations are not load-bearing. The reviewer concern about a missing naive-asynchronous-pipeline baseline is a legitimate evaluation-design limitation: without such a baseline one cannot attribute all of the speedup to the graph transformations, but that is a comparison-methodology gap, not circularity, because the measured gains are not true by construction. Overall, no step in the paper's derivation chain reduces to its own inputs, and the central claims rest on external measurements.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

HedraRAG is an engineering system rather than a mathematical derivation. Its central performance claims depend on several runtime-tuning parameters and on empirical workload assumptions measured on three QA datasets, namely semantic closeness of successive retrievals, token-level generation proximity, and skewed cluster access. No new physical entities are introduced; RAGraph is a software abstraction and speculative edges are scheduling devices.

free parameters (8)

mb, sub-stage time budget = not reported; selected via Eq. (1)
Controls how many retrieval clusters or generation steps form one sub-stage; the paper maximizes an estimated latency improvement, but the resulting value is not disclosed.
beta, scheduling overhead parameter = not reported
Appears in Eq. (1) as the CPU overhead of request scheduling and intermediate result handling; no measured value is given.
tau, speculative trigger threshold = not reported
Speculative execution starts when T_curr/T_max is below tau; the threshold is described as empirical but never quantified.
g_c, number of GPU-cached clusters = g_c = 4 in Figure 11
The size of the partial GPU index cache, trading generation memory against retrieval acceleration; the paper gives it only in a diagram.
GPU index cache update interval = every 50 sub-stages in practice
Sets how often hotspot clusters are swapped in and out; chosen to avoid PCIe contention but not derived from measurements shown in the paper.
g_s or KV_size, GPU memory split = selected by Eq. (2) from offline benchmarks; exact value not reported
Determines how much GPU memory goes to KV cache versus index clusters; the optimization depends on offline benchmarked throughput models.
local cache top-k for reordering = 20 in practice
Larger top-k results of a previous retrieval are cached for reuse; the value 20 is stated without a sensitivity study.
semantic drift threshold delta_s = not quantified
Used to choose which generation stage benefits most from speculative retrieval; no value or measurement procedure is given.

assumptions (6)

domain assumption Retrieval is CPU-bound and generation is GPU-bound, with the vector index in host memory and LLM weights and KV cache on the GPU.
Section 2.2 introduces this hardware model; all three optimizations depend on moving work between these resources.
domain assumption Generation and retrieval stages can be split into sub-stages without changing workflow semantics.
Section 4.2 models partitioning as node splitting; if sub-stage boundaries changed retrieval results or generation outputs, rollback-free operation would fail.
domain assumption Successive retrievals in a request are semantically close and partial generations approximate final embeddings.
Section 3.2 and Figure 7 justify speculative execution; the observations are measured only on wikiQA and hotpotQA.
domain assumption IVF index access is skewed and hotspot clusters persist across the 50-sub-stage cache update interval.
Section 3.3 and Figure 8, together with Section 4.4; partial GPU caching only helps if this skew persists at runtime.
domain assumption Retrieval requests arrive evenly across sub-stages, so the expected wait time can be modeled as mb/2 in Eq. (1).
Section 4.2 states this assumption explicitly; bursty arrivals would invalidate the time-budget formula.
domain assumption ANNS early termination and top-k result comparison preserve final answer quality.
Section 4.3 relies on overlap between partial and complete search results, with rollback to avoid quality loss.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving." pith.science (2026). https://pith.science/paper/4F6CW2T5

@misc{pith2026250709138,
  author       = {Pith},
  title        = {Pith review of: HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4F6CW2T5}},
  note         = {Machine review of arXiv:2507.09138}
}

read the original abstract

This paper addresses emerging system-level challenges in heterogeneous retrieval-augmented generation (RAG) serving, where complex multi-stage workflows and diverse request patterns complicate efficient execution. We present HedraRAG, a runtime system built on a graph-based abstraction that exposes optimization opportunities across stage-level parallelism, intra-request similarity, and inter-request skewness. These opportunities are realized through dynamic graph transformations, such as node splitting, reordering, edge addition, and dependency rewiring, applied to wavefronts of subgraphs spanning concurrent requests. The resulting execution plans are mapped onto hybrid CPU-GPU pipelines to improve resource utilization and reduce latency. Evaluations across a wide range of RAG workflows demonstrate speedups exceeding 1.5x and reaching up to 5x over existing frameworks, showcasing the effectiveness of coordinated generation and retrieval in serving environments.

Figures

Figures reproduced from arXiv: 2507.09138 by the authors.

**Figure 2.** The overview of HedraRAG, an LLM-Vector search co-designed system, to efficiently transform and schedule hetero [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Advanced RAG design and heterogeneous workflows, involving multi-round LLM-ANNS interaction. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: The comparison between different batching strate [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: Comparison of CPU-GPU pipeline efficiency with different strategies. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Workload variation in (a) LLM generation and (b) [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 8.** Figure 8: Skewness of access frequency among clusters when [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]

**Figure 9.** Figure 9: Opportunities from semantic similarity in RAG. (a) [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

**Figure 10.** Figure 10: Leveraging semantic similarity with speculative [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

**Figure 11.** Figure 11: Hybrid retrieval engine of HedraRAG with [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

**Figure 12.** Figure 12: Average request latency when using various RAG workflows. [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]

**Figure 14.** Figure 14: Average request latency with concurrency of dif [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]

**Figure 15.** Figure 15: Average request latency on larger LLM models, [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]

**Figure 16.** Figure 16: The fine-grained partitioning of HedraRAG to [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]

**Figure 17.** Figure 17: The speculation accuracy and latency comparison [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]

**Figure 18.** Figure 18: The speedups of GPU indexing and the hotspot [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Natural Language Query to Configuration for Retrieval Agents
cs.AI 2026-05 unverdicted novelty 6.0 of 10

BRANE maps queries to optimal retrieval pipeline configurations using LLM-derived features and per-configuration correctness predictors, improving the cost-quality Pareto frontier on three benchmarks.

Reference graph

Works this paper leans on

83 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

[n. d.]. Haystack. https://github.com/deepset-ai/haystack
[2]

[n. d.]. langChain. https://github.com/langchain-ai/langchain
[3]

Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. 2020. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics9, 8 (2020), 1295

2020
[4]

Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al. 2024. Open- scholar: Synthesizing scientific literature with retrieval-augmented lms.arXiv preprint arXiv:2411.14199(2024)

arXiv 2024
[5]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection.arXiv preprint arXiv:2310.11511(2023)

arXiv 2023
[6]

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bog- dan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. InInternational conference on machine learning. PMLR, 2206–2240

2022
[7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

2020
[8]

Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi, Yuexian Zou, and Shum- ing Shi. 2024. Retrieval is accurate generation.arXiv preprint arXiv:2402.17532 (2024)

arXiv 2024

Show all 83 references

[9]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions.arXiv preprint arXiv:1704.00051 (2017)

2017 arXiv
[10]

Leonardo Dagum and Ramesh Menon. 1998. OpenMP: an industry standard API for shared-memory programming.IEEE computational science and engineering5, 1 (1998), 46–55

1998
[11]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

2019
[12]

Yufei Ding, Lin Ning, Hui Guan, and Xipeng Shen. 2017. Generalizations of the theory and deployment of triangular inequality for compiler-based strength reduction. InProceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation. 33–48

2017
[13]

Yufei Ding, Xipeng Shen, Madanlal Musuvathi, Todd Mytkowicz, and Madan Musuvathi. 2015. Top: A framework for enabling algorithmic optimizations for distance-related problems. InPVLDB. 1046–1057

2015
[14]

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]

2024 arXiv
[15]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

2024 arXiv
[16]

Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels.arXiv preprint arXiv:2212.10496(2022)

2022 arXiv
[17]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997(2023)

2023 arXiv
[18]

In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems6 (2024), 325–338

2024
[19]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

2025 arXiv
[20]

Zijian He, Reyna Abhyankar, Vikranth Srivatsa, and Yiying Zhang. 2025. Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning.arXiv preprint arXiv:2502.08056(2025)

2025 arXiv
[21]

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060(2020)

2020 arXiv
[22]

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556(2022)

2022 arXiv
[23]

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al . 2023. Metagpt: Meta programming for multi-agent collaborative framework.arXiv preprint arXiv:2308.003523, 4 (2023), 6

2023 arXiv
[24]

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research24, 251 (...

2023
[25]

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park
[26]

Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.arXiv preprint arXiv:2310.06839 (2023)

2023 arXiv
[27]

Wenqi Jiang, Suvinay Subramanian, Cat Graves, Gustavo Alonso, Amir Yazdan- bakhsh, and Vidushi Dadu. 2025. RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving.arXiv preprint arXiv:2503.14649 (2025). 12 HedraRAG: Coordinating LLM Generation and ...

2025 arXiv
[28]

Wenqi Jiang, Marco Zeller, Roger Waleffe, Torsten Hoefler, and Gustavo Alonso
[29]

Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska
[30]

Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation.arXiv preprint arXiv:2305.06983(2023)

2023 arXiv
[31]

Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation.arXiv preprint arXiv:2404.12457(2024)

2024 arXiv
[32]

Piperag: Fast retrieval-augmented generation via algorithm-system co- design.arXiv preprint arXiv:2403.05676(2024)

2024 arXiv
[35]

Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024. FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research.arXiv preprint arXiv:2405.13576(2024)

2024 arXiv
[36]

Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06984(2023)

2023 arXiv
[37]

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel
[38]

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data7, 3 (2019), 535–547

2019
[39]

Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang
[40]

Mario Köppen. 2000. The curse of dimensionality. In5th online world conference on soft computing in industrial applications (WSC5), Vol. 1. 4–8

2000
[41]

InInterna- tional Conference on Machine Learning

Large language models struggle to learn long-tail knowledge. InInterna- tional Conference on Machine Learning. PMLR, 15696–15707
[42]

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

2020 arXiv
[43]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

2020
[44]

Tree of clarifications: Answering ambiguous questions with retrieval- augmented large language models.arXiv preprint arXiv:2310.14696(2023)

2023 arXiv
[45]

Jerry Liu. 2022. LlamaIndex. https://github.com/jerryjliu/llama_index. doi:10. 5281/zenodo.1234

2022
[46]

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

2019
[47]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

2023
[48]

ONNX Community. 2017. ONNX: Open Neural Network Exchange. https: //onnx.ai/. Accessed: 2025-05-27

2017
[49]

Chien-Yu Lin, Keisuke Kamahori, Yiyu Liu, Xiaoxiang Shi, Madhav Kashyap, Yile Gu, Rulin Shao, Zihao Ye, Kan Zhu, Stephanie Wang, et al. 2025. TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval. arXiv preprint arXiv:2502.20969(2025)

2025 arXiv
[50]

A Paszke. 2019. Pytorch: An imperative style, high-performance deep learning library.arXiv preprint arXiv:1912.01703(2019)

2019 arXiv
[51]

Thomas Merth, Qichen Fu, Mohammad Rastegari, and Mahyar Najibi. 2024. Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation.arXiv preprint arXiv:2404.06910(2024)

2024 arXiv
[52]

Rolf Neugebauer, Gianni Antichi, José Fernando Zazo, Yury Audzevich, Sergio López-Buedo, and Andrew W Moore. 2018. Understanding PCIe performance for end host networking. InProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 327–341

2018
[53]

Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Ganesh Ananthanarayanan, Ravi Netravali, and Junchen Jiang. 2024. RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation.arXiv preprint arXiv:2412.10543(2024)

2024
[54]

OpenAI. 2024. Learning to Reason with Language Models. https://openai.com/ index/learning-to-reason-with-llms/

2024
[55]

Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. InProceedings of the IEEE/CVF international conference on computer vision. 2998–3009

2023
[56]

Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024. Large language model based long-tail query rewriting in taobao search. InCompanion Proceedings of the ACM on Web Conference 2024. 20–28

2024
[57]

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models.Transactions of the Association for Computational Linguistics11 (2023), 1316–1331

2023
[58]

A Vaswani. 2017. Attention is all you need.Advances in Neural Information Processing Systems(2017)

2017
[59]

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.arXiv preprint arXiv:2305.15294(2023)

2023 arXiv
[60]

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents.Frontiers of Computer Science18, 6 (2024), 186345

2024
[61]

Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards End-to-End Optimization of LLM-based Applications with Ayo. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1302–1316

2025
[62]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

2023 arXiv
[63]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

2022
[64]

Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xi- angyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. 2021. Milvus: A purpose-built vector data management system. InProceedings of the 2021 International Conference on Management of Data. 2614–2627

2021
[65]

Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024. RECOMP: Improving retrieval- augmented LMs with context compression and selective augmentation. InThe Twelfth International Conference on Learning Representations

2024
[66]

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533(2022)

2022 arXiv
[67]

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672(2024)

2024 arXiv
[68]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for di- verse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600 (2018)

2018 arXiv
[69]

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2025. The rise and potential of large language model based agents: A survey.Science China Information Sciences 68, 2 (2025), 121101

2025
[70]

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly.High-Confidence Computing(2024), 100211

2024
[71]

Qian Xu, Juan Yang, Feng Zhang, Junda Pan, Kang Chen, Youren Shen, Amelie Chi Zhou, and Xiaoyong Du. 2025. Tribase: A Vector Data Query Engine for Reliable and Lossless Pruning Compression using Triangle Inequalities.Proceedings of the ACM on Management of Data3, 1 (2025), 1–28

2025
[72]

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation.arXiv preprint arXiv:2401.15884(2024)

2024 arXiv
[73]

Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. 2024. Infer- ence scaling for long-context retrieval augmented generation.arXiv preprint arXiv:2410.04343(2024)

2024 arXiv
[74]

Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. InProceedings of the Twentieth European Conference on Computer Systems. 94–109

2025
[75]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068 (2022). 13 SOSP ’25, October 13–16, 2025, Se...

2022 arXiv
[76]

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538

2022
[77]

Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llms.arXiv preprint arXiv:2407.02485(2024)

2024 arXiv
[78]

Zhihao Zhang, Alan Zhu, Lijie Yang, Yihua Xu, Lanting Li, Phitchaya Mangpo Phothilimthana, and Zhihao Jia. 2024. Accelerating retrieval-augmented language model serving with speculation.arXiv preprint arXiv:2401.14021(2024)

2024 arXiv
[79]

Huan Zhang, Yu Song, Ziyu Hou, Santiago Miret, and Bang Liu. 2024. Honey- comb: A flexible llm-based agent system for materials science.arXiv preprint arXiv:2409.00135(2024)

2024 arXiv
[80]

Justin Zobel and Alistair Moffat. 2006. Inverted files for text search engines.ACM computing surveys (CSUR)38, 2 (2006), 6–es. 14

2006
[81]

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al . 2023. Siren’s song in the AI ocean: a survey on hallucination in large language models.arXiv preprint arXiv:2309.01219(2023)

2023 arXiv
[82]

Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, and Xin Jin. 2024. Fast Vector Query Processing for Large Datasets Beyond{GPU} Memory with Re- ordered Pipelining. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 23–40

2024
[84]

Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond Yes and No: Improving Zero-Shot Pointwise LLM Rankers via Scoring Fine-Grained Relevance Labels. InProceedings of the 2024 Conference of the North American Chapter of the Ass...

2024
[2023]

Chameleon: a heterogeneous and disaggregated accelerator system for retrieval-augmented language models.arXiv preprint arXiv:2310.09949(2023)

2023 arXiv
[2024]

Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity.arXiv preprint arXiv:2403.14403(2024)

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

[n. d.]. Haystack. https://github.com/deepset-ai/haystack

[2] [2]

[n. d.]. langChain. https://github.com/langchain-ai/langchain

[3] [3]

Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. 2020. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics9, 8 (2020), 1295

2020

[4] [4]

Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al. 2024. Open- scholar: Synthesizing scientific literature with retrieval-augmented lms.arXiv preprint arXiv:2411.14199(2024)

arXiv 2024

[5] [5]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection.arXiv preprint arXiv:2310.11511(2023)

arXiv 2023

[6] [6]

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bog- dan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. InInternational conference on machine learning. PMLR, 2206–2240

2022

[7] [7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

2020

[8] [8]

Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi, Yuexian Zou, and Shum- ing Shi. 2024. Retrieval is accurate generation.arXiv preprint arXiv:2402.17532 (2024)

arXiv 2024