Pith. sign in

REVIEW 3 major objections 5 minor 76 references

RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RAGDoll makes RAG serving on a single consumer GPU practical by decoupling CPU retrieval and GPU generation into parallel pipelines with joint memory placement, cutting average latency by up to 3.6× versus serial RAG baselines.

desk verdict Credible integrated systems work with a plausible speedup claim, but the pipelining benefit itself is not cleanly isolated. read the letter →

arxiv 2504.15302 v1 pith:JTNXVB43 submitted 2025-04-17 cs.DC cs.OS

Weiping Yu , Ningyi Liao , Siqiang Luo , Junfeng Liu This is my paper · ORCID
classification cs.DCcs.OS
keywords RAGservingLLMoffloadingpipelineparallelismbatchschedulingmemoryhierarchysingle-GPUinferencevectordatabaseadaptiveconfiguration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAGDoll is built on the observation that a RAG request spends much of its time waiting: the CPU does retrieval, then the GPU does generation, and in a serial workflow whichever device is not working sits idle. The paper's central claim is that on a single consumer GPU these two stages can be decoupled into parallel pipelines with coordinated memory placement, so the same hardware serves 8B-70B models with 1.9× to 3.6× lower average latency than serial RAG systems, and up to 11.7× lower under severe resource constraints. The system also adapts batch sizes to the online backlog so that slow retrieval does not force the GPU to wait for the next batch. If correct, this means a 12-24GB GPU plus 176-256GB of main memory is enough for practical small-scale RAG deployment, a regime that current RAG serving frameworks largely ignore.

What carries the argument

The load-bearing mechanism is the two-worker multi-pipeline: retrieval and generation run as decoupled queues, so batches of different sizes can move independently and one stage can prefetch while the other computes. This is coordinated by joint hierarchical memory placement across GPU, CPU, and disk, and by a backlog-aware batch scheduler that chooses batch sizes through the cost model $T(B)=aB^c$, with the rule that maximum batch size minimizes average latency only when processing time scales sublinearly enough (for a two-way split, roughly $c \leq \log_2(3/2) \approx 0.585$). An offline active-profiling step balances retrieval and generation latencies ahead of time, and an online scheduler adjusts batch size and memory placement as request backlogs change.

What would settle it

On the paper's low-end platform, instrument the retrieval and generation pipelines separately while serving the 8B TriviaQA workload; if an overlapped batch pair takes nearly the sum of the two standalone durations, or if the average-latency speedup over the serial RAG baseline falls below roughly 1.5×, the concurrency premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the latency of memory-constrained RAG serving is dominated neither by retrieval nor by generation alone, but by idle time and resource contention between them: in a serial execution, the CPU and GPU each wait while the other works, and fixed memory placement forces both components to compete for the same RAM. RAGDoll's discovery is that treating retrieval and generation as separate workers, each with its own batch queue and its own memory footprint, lets them overlap: the retrieval worker pulls database partitions from disk into RAM and formats generation batches while the GPU worker prefetches LLM tensors and generates tokens, while a joint memory manager moves database partitions, LLM weights, and KV caches between GPU, CPU, and disk according to a profiled configuration. On top of this, a backlog-aware batch scheduler uses an empirical cost curve $T(B)=aB^c$ to pick the batch size that minimizes average latency under the current request rate. In experiments with a large question-answering knowledge base and 8B/70B models, this design cuts average latency by up to 3.6× against a serial RAG baseline and by up to 11.7× under constrained memory, with idle time dropping from roughly 80% to 30% of the workflow.

Load-bearing premise

The argument assumes that CPU-side retrieval and GPU-side generation can run at the same time without materially slowing each other down, so that the overlap gains are not eaten by contention.

Editorial extensions

If this is right

  • A single 12-24GB GPU with 176-256GB of host memory can serve 8B and 70B RAG workloads under dynamic arrival rates, so resource-constrained deployments no longer have to choose between retrieval quality and model size.
  • Because retrieval and generation workers use independent batch queues, retrieval can batch many queries while generation batches stay small, removing the forced coupling that makes serial RAG systems suboptimal.
  • CPU idle time in the RAG workflow drops from roughly 80% to 30%, and CPU utilization roughly doubles, converting waiting hardware into useful work.
  • Average end-to-end latency falls by 1.9× to 3.6× against a serial RAG baseline and by up to 11.7× under constrained memory, with waiting time reduced by up to 20× and generation time by up to 5×.
  • The scheduler adapts batch size and memory placement to arrival-rate changes without prior knowledge of the workload, as shown by policy shifts from batch size 16 to 48 under rising load.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the decoupled-pipeline argument should transfer to other retrieval-side costs such as reranking, filtering, and re-embedding, and to multi-tenant RAG serving on one GPU, since the same idle-time mechanism applies.
  • The batch-splitting inequality suggests a portable tuning rule for any offloading RAG system: measure the exponent $c$ in $T(B)=aB^c$ on the target hardware and prefer smaller batches when $c$ exceeds the threshold implied by the split count; the paper does not evaluate this rule outside its own scheduler.
  • Because the reported gains come mainly from absorbing backlog, a workload with perfectly smooth, low request rates should show much smaller speedups; that boundary is not reported.
  • Editorial inference: on hardware with slower PCIe or disk I/O than the tested platforms, the overlap between retrieval and generation may shrink, so the 3.6× figure should be read as a ceiling for the tested class of hardware, not a universal guarantee.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents RAGDoll, a system for online RAG serving on a single GPU with limited memory. It contributes three mechanisms: a joint memory-placement scheme coordinating vector-database partitions, LLM weights, and KV caches across GPU/CPU/disk; a multi-pipeline design that decouples CPU-side retrieval from GPU-side generation and prefetches LLM tensors asynchronously; and a two-step adaptive configuration scheme with offline active profiling and online backlog-aware batch scheduling. The evaluation compares RAGDoll against serial RAG integrations built on vLLM and Hugging Face Accelerate on two platforms (PF-High with A30, PF-Low with A5000) with 8B and 70B Llama models under a synthetic Poisson workload, reporting up to 3.6x average-latency speedup over vLLM-based RAG and up to 11.7x over Accelerate-based RAG. The implementation is a 5,000-line prototype integrating FlexGen and Milvus.

Significance. If the result holds, RAGDoll addresses a practically important problem: serving large RAG workloads on a single 12-24GB GPU. The paper has genuine strengths: it implements the full system, ablates each design component, evaluates on two platforms and two model scales, and reports a component-wise latency breakdown. However, the central mechanism—that retrieval and generation can be overlapped with negligible interference—is neither isolated nor measured, and the only quantitative evidence for the headline speedup comes from a single run of a synthetic workload. The contribution is therefore promising but not yet established.

major comments (3)
  1. [§6.3, Tables 1-2] The load-bearing premise of the multi-pipeline design is that CPU-side retrieval and GPU-side generation can run concurrently with only minor mutual interference, yet no experiment isolates interference. Table 1 shows that RAGDoll's per-request retrieval+generation time is not lower than vLLMRAG's (386s vs 350s on PF-Low 8B), and Section 6.3 itself concedes that 'pipelined retrieval and generation processes may interfere with each other.' The 'Without pipelined design' ablation in Table 2 removes the overlap, but it simultaneously forces retrieval and generation to share a batch size and memory policy, so the 38% and 58% penalties cannot be attributed to losing overlap alone. Please add an experiment that keeps RAGDoll's independent batch-size and memory policies but serializes the two stages, or report device-level counters such as PCIe bandwidth, CPU memory bandwidth, and disk I/O during overlap, so the reader can see how much of the speedup comes from overlap rather than from separate batch scheduling.
  2. [§4.4, Eq. (5)-(8)] The derivation of the optimal-batch-size condition is not mathematically sound as written. In Eq. (5), L1 = T(n) - (1/n) Σ t_i omits the batch start time and the definition of t_i is missing; as written the quantity can be negative when arrival times are large, and it is not obviously a per-request latency. Eq. (6) introduces a (k+1)/2 factor without derivation. Because the online scheduler uses this formula, with a and c fitted from profiling samples, to choose between batch sizes, the claim that the maximum batch size is optimal for c below a threshold is not established by the text. Please either provide a correct derivation with explicitly defined arrival and completion times, or state explicitly that Eqs. (5)-(8) constitute a heuristic and validate it empirically against measured latencies for different batch splits.
  3. [§6.1, Figure 7] The headline speedup rests on a single realization of a stochastic workload. Section 6.1 states that request arrivals are simulated with a Poisson distribution with varying rates, but the evaluation reports no seeds, no repeated trials, and no error bars or confidence intervals. Under a Poisson process, backlog dynamics vary from run to run, and the 1.9x-3.6x figures could be partly due to a favorable draw. Please report mean and variance over multiple seeds for the main end-to-end experiments, and state whether the speedup is consistent across runs. This is not a presentation issue: the central quantitative claim is at stake.
minor comments (5)
  1. [§6.1] The two platforms use A30 and A5000 GPUs, which are professional/datacenter-class rather than consumer-grade devices; the paper's 'consumer-grade' framing should be qualified or supplemented with results on a mainstream consumer GPU (e.g., RTX 30/40 series), or at least with a caveat about PCIe and host-memory-bandwidth differences.
  2. [Figure 7 caption] The caption says that 'the linear segments between two distinct turning points represent a generation batch,' but the connection between the slope and the batch size is not explained; please clarify how batch boundaries are read off the figure.
  3. [References [2] and [3]] References [2] and [3] both point to arXiv 2407.07000 and appear to describe the same work; please disambiguate or remove the duplicate.
  4. [§5, Evaluation setup] The implementation and experimental artifacts are not released; for a systems paper whose evidence is a custom 5,000-line prototype, an artifact or appendix with code and configuration details would substantially aid reproducibility.
  5. [Table 2] The table caption says that the gray number is the static generation batch size policy, but it is unclear what the gray values in parentheses signify for each row; please define them explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: RAGDoll's speedup is an empirical systems result, and the profiling-based batch scheduling rule is a control heuristic rather than a hidden replay of the headline claim.

full rationale

The paper's central claim is an empirical end-to-end comparison: RAGDoll is measured against serial RAG systems built on vLLM and Accelerate (Section 6.2), with the 1.9x-3.6x speedups reported as observed average latencies under the same dynamic workload. The only derivation-like content is the batch scheduling analysis in Section 4.4, where the paper assumes a power-law model T(B)=aB^c (Eq. 4) and derives the threshold 2k^c <= k+1 (Eq. 7) for when the maximum batch size minimizes average latency. The constants a and c are fitted from profiling samples, and the rule is used to select batch sizes at runtime; this is a fitted control policy, not an independent prediction that is then presented as the paper's main result. The headline speedups are not computed from Eq. (4)-(8) but from actual latency measurements against external baselines, so nothing reduces by construction to the model's inputs. There are no load-bearing self-citations: the cited systems (FlexGen, Milvus, vLLM, RAGCache, CacheBlend, etc.) are external prior work, and the paper explicitly states that 'pipelining retrieval and generation is not new' (Section 4.2). The Section 6.3 concession that 'pipelined retrieval and generation processes may interfere with each other' is an honest limitation and a correctness risk about unmeasured interference, not an instance of circular reasoning. Likewise, the ablation in Table 2 is confounded (removing pipelining also removes independent batch sizing), but confounding is an experimental-design issue, not circularity. Overall, the performance claims are self-contained and externally benchmarked, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No invented physical entities are introduced. The performance claim rests on fitted power-law latency parameters and on empirical assumptions about partition-loading dominance and low CPU-GPU interference. These assumptions are not independently verified in the paper.

free parameters (1)
  • latency scaling coefficients a and c in T(B)=aB^c = not reported
    Equation (4) models batch processing time as a power law. The paper says runtime evaluation uses samples from active profiling to fit this model and choose batch sizes; the fitted values are not listed.
assumptions (3)
  • domain assumption Retrieval latency for a fixed number of resident partitions is nearly constant across retrieval batch sizes because partition loading dominates.
    Section 4.4 uses this to reduce offline profiling to the generation batch size; if false, the configuration search could miss better retrieval batching.
  • domain assumption CPU-bound retrieval and GPU-bound generation can be overlapped without destructive contention.
    This is the basis of the multi-pipeline design; Section 6.3 notes possible interference but does not measure it.
  • ad hoc to paper Batch processing time follows the power law T(B)=aB^c with constant a,c during a workload interval.
    Equation (4) is introduced for the scheduling rule and the coefficients are fitted from profiling samples rather than derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU." pith.science (2026). https://pith.science/paper/JTNXVB43

@misc{pith2026250415302,
  author       = {Pith},
  title        = {Pith review of: RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTNXVB43}},
  note         = {Machine review of arXiv:2504.15302}
}
read the original abstract

Retrieval-Augmented Generation (RAG) enhances large language model (LLM) generation quality by incorporating relevant external knowledge. However, deploying RAG on consumer-grade platforms is challenging due to limited memory and the increasing scale of both models and knowledge bases. In this work, we introduce RAGDoll, a resource-efficient, self-adaptive RAG serving system integrated with LLMs, specifically designed for resource-constrained platforms. RAGDoll exploits the insight that RAG retrieval and LLM generation impose different computational and memory demands, which in a traditional serial workflow result in substantial idle times and poor resource utilization. Based on this insight, RAGDoll decouples retrieval and generation into parallel pipelines, incorporating joint memory placement and dynamic batch scheduling strategies to optimize resource usage across diverse hardware devices and workloads. Extensive experiments demonstrate that RAGDoll adapts effectively to various hardware configurations and LLM scales, achieving up to 3.6 times speedup in average latency compared to serial RAG systems based on vLLM.

Figures

Figures reproduced from arXiv: 2504.15302 by the authors.

Figure 1
Figure 1. Representative techniques related to RAG serving, depicted by corresponding tasks and design objectives. In general, a RAG workflow first retrieves data from the knowledge database according to user requests, then gen￾erates responses by the LLM based on the extracted infor￾mation. The integration of retrieval-based techniques, how￾ever, poses new challenges for comprehensive performance enhancements, especially as … view at source ↗
Figure 2
Figure 2. Pipelines in memory-intense RAG systems: (a) Standard overlapping LLM inference may misalign compu￾tation and prefetching due to CPU scheduling and compute jitter. (b) Our LLM pipeline separates computation and com￾munication for continuous prefetching. (c) Fixed batch sched￾uling accumulates larger backlogs under memory-intense conditions. (d) Our backlog-aware batch scheduling adjusts flexibly to minimize backlogs… view at source ↗
Figure 3
Figure 3. Dissecting an online offloading-based RAG system. (a) LLM tensor placement. (b) vector database residents. (c) compute workspace scheduling. latency. RAGCache implements a maximum batch size ap￾proach to pursue lower average latency, while CacheBlend evaluates performance across different batch scheduling without specifying batch sizes for on-demand scenarios. These approaches demonstrate that effective batching tec… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: CPU and GPU utilization and memory usage vary in a serial retrieval and generation mode when using different batch sizes under a static memory allocation policy. long idle times waiting for the computation on the other one, undermining system throughput and efficiency.…
Figure 6
Figure 6. Figure 6: Timelines of computation and memory operations in RAGDoll: (a) memory placement for LLM weights and KV cache; (b) computation workspace of multi-pipeline; (c) memory placement for database partitions. and acquisition, executing these operations between consec￾utive bat…
Figure 7
Figure 7. Figure 7: Request latency under a dynamic workload. On the x-axis, the arrival rate increases from left to right: approximately 0–80 indicates 4 requests per minute, 80–240 corresponds to 8 requests per minute, 240–480 to 12 requests per minute, and 480–800 to 16 requests per mi…
Figure 8
Figure 8. Figure 8: Boxplot of all latency values. We integrate the baseline LLM servings with RAG in a serial mode, processing batches in their arrival order with an adaptive size 4𝜆(𝑡) for each interval. On PF-Low, both baselines encounter memory limitations when serving the 70B model, …
Figure 10
Figure 10. Figure 10: Average latency with different top-k values in retrieval. The “X” mark indicates extreme high latency due to the method’s poor performance. batch sizes, while RAGDoll relies on its pipeline architec￾ture and balanced configuration to mitigate the increase [PITH_FULL_…
Figure 11
Figure 11. Figure 11: Performance of 70B model under DiskANN. 6.5 Case Study Number of Retrieval Chunks. We evaluate the impact of different top-k chunks in retrieval. Changes in retrieval chunk count do not significantly affect retrieval latency be￾cause loading costs substantially outwei…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 23 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al . 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [3]

    Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, and Alexey Tumanov

    work page

  3. [4]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: In- ternational Conference for High Performance Computing, Networking, Storage and Analys...

    work page 2022

  4. [5]

    arXiv e-prints (2024), arXiv–2407

    Metron: Holistic performance evaluation framework for llm inference systems. arXiv e-prints (2024), arXiv–2407

    work page 2024

  5. [6]

    Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. 2020. ANN-Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems 87 (2020), 101374

    2020

  6. [7]

    Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts). 41–46

    work page 2023

  7. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al . 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  8. [9]

    Jordi Bayarri-Planas, Ashwin Kumar Gururajan, and Dario Garcia- Gasulla. 2025. Pareto-Optimized Open-Source LLMs for Healthcare via Context Retrieval. arXiv:2409.15127 [cs.AI] https://arxiv.org/abs/ 2409.15127

    arXiv 2025

Show all 76 references
  1. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  2. [11]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Bench- marking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 17754–17762

    2024

  3. [12]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

  4. [13]

    Chroma. 2025. https:// https://www.trychroma.com/

    2025

  5. [14]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hos- seini, and Hervé Jégou. 2024. The faiss library. arXiv preprint arXiv:2401.08281 (2024)

    2024 arXiv

  6. [15]

    Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, and Chun Jason Xue. 2025. FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference. In Proceedings of the 5th Workshop on Machine Learning and Systems . 56–65

    2025

  7. [16]

    Etienne Dilocker, Bob van Luijt, Byron Voorbach, Mohd Shukri Hasan, Abdel Rodriguez, Dirk Alexander Kulawiak, Marcin Antas, and Parker Duckworth. [n. d.]. Weaviate. https://github.com/weaviate/weaviate If you use this software, please cite it as below

  8. [17]

    Tianyu Fan, Jingyuan Wang, Xubin Ren, and Chao Huang. 2025. Mini- RAG: Towards Extremely Simple Retrieval-Augmented Generation. arXiv preprint arXiv:2501.06713 (2025)

    2025 arXiv

  9. [18]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    2022 arXiv

  10. [19]

    Artyom Eliseev and Denis Mazur. 2023. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238 (2023)

    2023 arXiv

  11. [20]

    Github. 2022. https://github.com/features/copilot

    2022

  12. [21]

    Siddharth Gollapudi, Neel Karia, Varun Sivashankar, Ravishankar Kr- ishnaswamy, Nikit Begwani, Swapnil Raz, Yiyong Lin, Yin Zhang, Nee- lam Mahapatro, Premkumar Srinivasan, et al. 2023. Filtered-diskann: Graph algorithms for approximate nearest neighbor search with filters. In...

    2023

  13. [22]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval- augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2 (2023)

    2023 arXiv

  14. [23]

    HuggingFace Accelerate. 2022. https:// hugging- face.co/docs/accelerate/index

    2022

  15. [24]

    Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. 2019. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Ad- vances in neural information processing Systems 32 (2019)

    2019

  16. [25]

    Google. 2023. https://bard.google.com/

    2023

  17. [26]

    Wenqi Jiang, Marco Zeller, Roger Waleffe, Torsten Hoefler, and Gus- tavo Alonso. 2023. Chameleon: a heterogeneous and disaggregated accelerator system for retrieval-augmented language models. arXiv preprint arXiv:2310.09949 (2023)

    2023 arXiv

  18. [27]

    Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska. 2024. Piperag: Fast retrieval-augmented generation via algorithm-system co-design. arXiv preprint arXiv:2403.05676 (2024)

    2024 arXiv

  19. [28]

    Wenqi Jiang, Suvinay Subramanian, Cat Graves, Gustavo Alonso, Amir Yazdanbakhsh, and Vidushi Dadu. 2025. RAGO: Systematic Perfor- mance Optimization for Retrieval-Augmented Generation Serving. arXiv preprint arXiv:2503.14649 (2025)

    2025 arXiv

  20. [29]

    Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. Advances in Neural Information Processing Systems 37 (2024), 59532–59569

    2024

  21. [30]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. Ragcache: Efficient knowledge caching for retrieval- augmented generation. arXiv preprint arXiv:2404.12457 (2024)

    2024 arXiv

  22. [31]

    Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. 2024. Neo: Saving gpu memory crisis with cpu offloading for online llm inference. arXiv preprint arXiv:2411.01142 (2024)

    2024 arXiv

  23. [32]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  24. [33]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  25. [34]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer

  26. [35]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al . 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing ...

    2020

  27. [36]

    Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110 (2022)

    2022 arXiv

  28. [37]

    LlamaIndex. 2022. https://www.llamaindex.ai/

    2022

  29. [38]

    LMSYS ORG. 2023. Chatbot Arena Leaderboard Week 8: Introduc- ing MT-Bench and Vicuna-33B. https://lmsys.org/blog/2023-06-22- leaderboard/

    2023

  30. [39]

    LangChain. 2022. https://github.com/langchain-ai/langchain

    2022

  31. [40]

    Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Augmented large language models with parametric knowledge guiding. arXiv preprint arXiv:2305.04757 (2023)

    2023 arXiv

  32. [41]

    Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836

    2018

  33. [42]

    Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553 (2020)

    2020 arXiv

  34. [43]

    Milvus. 2022. https://github.com/milvus-io/milvus

    2022

  35. [44]

    Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang

  36. [45]

    arXiv preprint arXiv:2410.07590 (2024)

    TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text. arXiv preprint arXiv:2410.07590 (2024)

    2024 arXiv

  37. [46]

    Bowen Pang, Kai Li, and Feifan Wang. 2025. Optimizing LLM Infer- ence Throughput via Memory-aware and SLA-constrained Dynamic Batching. arXiv preprint arXiv:2503.05248 (2025)

    2025 arXiv

  38. [47]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132

    2024

  39. [48]

    Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. One queue is all you need: Resolving head-of-line blocking in large language model serving. arXiv preprint arXiv:2407.00...

    2024 arXiv

  40. [49]

    Pinecone Systems, Inc. 2025. Pinecone. https://www.pinecone.io/. Accessed: 2025-04-03

    2025

  41. [50]

    John X Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexan- der M Rush. 2023. Text embeddings reveal (almost) as much as text. arXiv preprint arXiv:2310.06816 (2023)

    2023 arXiv

  42. [51]

    OpenAI. 2022. https://openai.com/blog/chatgpt

    2022

  43. [52]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval- augmented language models. Transactions of the Association for Com- putational Linguistics 11 (2023), 1316–1331

    2023

  44. [53]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

  45. [54]

    Aditi Singh, Suhas Jayaram Subramanya, Ravishankar Krishnaswamy, and Harsha Vardhan Simhadri. 2021. Freshdiskann: A fast and accurate graph-based ann index for streaming similarity search. arXiv preprint arXiv:2105.09613 (2021)

    2021 arXiv

  46. [55]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 590–606

    2024

  47. [56]

    Sonal Prabhune and Donald J Berndt. 2024. Deploying Large Lan- guage Models With Retrieval Augmented Generation. arXiv preprint arXiv:2411.11895 (2024)

    2024 arXiv

  48. [57]

    Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexan- dropoulos, Lahav Lipson, et al . 2024. Infinigen indoors: Photoreal- istic indoor scenes using procedural generation. In Proceedings of the IEEE/CVF ...

    2024

  49. [58]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  50. [59]

    Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al

  51. [60]

    In International Conference on Machine Learning

    Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning. PMLR, 31094–31116

  52. [61]

    Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, et al. 2024. Speculative rag: Enhancing retrieval aug- mented generation through drafting. arXiv preprint arXiv:2407.08223 (2024)

    2024 arXiv

  53. [62]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles . 640–654

    2024

  54. [63]

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2024. Dynamollm: Designing llm inference clusters for performance and energy efficiency. arXiv preprint arXiv:2408.00741 (2024)

    2024

  55. [64]

    Yifan Tan, Cheng Tan, Zeyu Mi, and Haibo Chen. 2024. PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption. arXiv preprint arXiv:2411.03357 (2024)

    2024 arXiv

  56. [65]

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems 35 (2022), 27168–27183

    2022

  57. [66]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 521–538. 14 RAGDoll: Efficient...

    2022

  58. [67]

    Xuanlei Zhao, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, and Yang You. 2024. Hetegen: Heterogeneous parallel inference for large language models on resource-constrained devices. arXiv preprint arXiv:2403.01164 (2024)

    2024 arXiv

  59. [68]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, et al. 2024. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems. arXiv preprint arXiv:2401.17644 (2024)

    2024 arXiv

  60. [69]

    Yun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko, Yinxiao Liu, Chu-Cheng Lin, Lei Shu, Liangchen Luo, Lei Meng, Bang Liu, et al. 2024. Acceler- ating inference of retrieval-augmented generation via sparse context selection. arXiv preprint arXiv:2405.16178 (2024). 15

    2024 arXiv

  61. [71]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning. PMLR, 38087–38099

    2023

  62. [72]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2024. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. arXiv preprint arXiv:2405.16444 (2024)

    2024 arXiv

  63. [76]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Syste...

    2024

  64. [2017]

    arXiv preprint arXiv:1705.03551 (2017)

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 (2017)

    2017 arXiv

  65. [2021]

    In Proceedings of the 2021 International Conference on Management of Data

    Milvus: A purpose-built vector data management system. In Proceedings of the 2021 International Conference on Management of Data. 2614–2627

    2021

  66. [2022]

    int8 (): 8-bit matrix multiplication for transformers at scale

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems 35 (2022), 30318–30332

    2022

  67. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles. 611–626

  68. [2024]

    arXiv preprint arXiv:2407.07000 (2024)

    Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems. arXiv preprint arXiv:2407.07000 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.