Pith. sign in

REVIEW 32 cited by

RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.12457 v2 pith:JRNDNKXZ submitted 2024-04-18 cs.DC cs.CLcs.LG

Chao Jin , Zili Zhang , Xuanlin Jiang , Fangyue Liu , Xin Liu , Xuanzhe Liu , Xin Jin This is my paper · ORCID
classification cs.DCcs.CLcs.LG
keywords knowledgeragcachecachinggenerationinferencefaissintermediatelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) has shown significant improvements in various natural language processing tasks by integrating the strengths of large language models (LLMs) and external knowledge databases. However, RAG introduces long sequence generation and leads to high computation and memory costs. We propose RAGCache, a novel multilevel dynamic caching system tailored for RAG. Our analysis benchmarks current RAG systems, pinpointing the performance bottleneck (i.e., long sequence due to knowledge injection) and optimization opportunities (i.e., caching knowledge's intermediate states). Based on these insights, we design RAGCache, which organizes the intermediate states of retrieved knowledge in a knowledge tree and caches them in the GPU and host memory hierarchy. RAGCache proposes a replacement policy that is aware of LLM inference characteristics and RAG retrieval patterns. It also dynamically overlaps the retrieval and inference steps to minimize the end-to-end latency. We implement RAGCache and evaluate it on vLLM, a state-of-the-art LLM inference system and Faiss, a state-of-the-art vector database. The experimental results show that RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 32 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCBench: A KV Cache-Centric Analysis of Long-Context Methods

    cs.CL 2024-12 conditional novelty 7.0 of 10

    A KV-cache-centric benchmark shows that long-context methods with sub-linear memory degrade sharply in multi-turn and multi-request reuse, while sparse-encoding methods with full memory remain robust.

  2. Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.

  3. FinCacheServe: Dependency-Consistent Answer Reuse for Cost-Efficient RAG Serving over Mutable Enterprise Documents

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Dependency-checked answer reuse lets FinCacheServe skip about 53% of LLM calls in hosted SEC-derived RAG traces without serving any observed stale outputs.

  4. Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents

    cs.PL 2026-06 unverdicted novelty 6.0 of 10

    FCGraft synthesizes code policies for embodied agents by grafting KV caches from a library of validated functions, claiming 18.31% higher success rate and 2.3x faster synthesis than prompt-level caching.

  5. Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulation

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A latent-space reasoning cache with a lightweight decoder cuts the cost of LLM-based human mobility simulation by roughly 40-90% while keeping trajectory quality comparable.

  6. Strata: Hierarchical Context Caching for Long Context Language Model Serving

    cs.DC 2025-08 conditional novelty 6.0 of 10

    Strata uses GPU-assisted I/O and cache-aware scheduling to cut the cost of loading cached KV states, raising long-context serving throughput by up to 5x at equal latency.

  7. HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving

    cs.DB 2025-07 conditional novelty 6.0 of 10

    HedraRAG uses a graph abstraction and dynamic transformations to pipeline generation and retrieval stages, achieving 1.5x to 5x speedups in heterogeneous RAG serving.

  8. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows

    cs.DC 2025-07 conditional novelty 6.0 of 10

    KVFlow uses workflow-aware eviction priorities and overlapped KV prefetching to cut cache-miss latency in LLM multi-agent serving.

  9. Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents

    cs.AI 2025-05 conditional novelty 6.0 of 10

    SearchAgent-X speeds up LLM-based search agents by prioritizing requests that benefit most from cached computations and by stopping approximate retrieval early when the model is ready, with no measured loss in answer quality.

  10. SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cache

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SubGCache reuses precomputed KV caches at the subgraph level for batches of graph RAG queries, cutting TTFT by up to 6.68x with comparable accuracy.

  11. PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications

    cs.DC 2025-05 conditional novelty 6.0 of 10

    PrefillOnly serves single-token LLM requests with hybrid prefilling and JCT-aware scheduling, achieving up to 4x higher throughput on long-input workloads.

  12. Guillotine: Hypervisors for Isolating Malicious AIs

    cs.CR 2025-04 conditional novelty 6.0 of 10

    Guillotine is a proposed hypervisor architecture that sandboxes dangerous AI models through dedicated hardware cores, monitored I/O ports, physical fail-safes like cable disconnection, and regulatory enforcement.

  13. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Cache-Craft reuses stored attention states for repeated RAG text chunks with selective recomputation, cutting redundant LLM prefill computation by 51% over prefix caching while retaining roughly 90% of answer quality.

  14. LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

    cs.LG 2024-11 conditional novelty 6.0 of 10

    LLMSteer improves long-context question answering quality and speed by steering attention toward tokens that rank highly across two differently prompted readings of the same context.

  15. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    cs.LG 2026-07 conditional novelty 5.0 of 10

    KAP compiles structured knowledge priors into runtime access plans that cut proposal-time KV access to 5.5% at 128K while matching full-context QA quality and yielding up to 1.19× decode throughput.

  16. ConceptBot: Enhancing Robot's Autonomy through Task Decomposition with Large Language Models and Knowledge Graph

    cs.RO 2025-08 conditional novelty 5.0 of 10

    Using ConceptNet-augmented prompts, ConceptBot reports 87% vs 31% success on implicit tasks and 76% vs 15% on risk-aware tasks over a re-implemented SayCan baseline, with an 80% SafeAgentBench score.

  17. Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding

    cs.CL 2025-07 conditional novelty 5.0 of 10

    DeepSeek-R1-distilled models show higher multi-document QA accuracy than their base counterparts and flatter position-bias curves, especially with 50-80 documents.

  18. WebANNS: Fast and Efficient Approximate Nearest Neighbor Search in Web Browsers

    cs.IR 2025-07 conditional novelty 5.0 of 10

    A new browser-native ANNS engine uses WebAssembly, lazy loading, and a memory optimizer to cut P99 query latency by up to 743.8x and memory use by up to 39% compared to Mememo.

  19. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  20. Efficient Vector Search on Disaggregated Memory with d-HNSW

    cs.DB 2025-05 reject novelty 5.0 of 10

    A disaggregated RDMA-based HNSW design using a small cached routing index and batched cluster fetches, with evaluation only against self-defined baselines.

  21. RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU

    cs.DC 2025-04 conditional novelty 5.0 of 10

    RAGDoll pipelines retrieval and generation, jointly manages memory across disk, RAM, and GPU, and adaptively sizes batches to cut average RAG latency by up to 3.6x on a single GPU.

  22. GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines

    cs.NI 2025-01 conditional novelty 5.0 of 10

    The paper proposes offloading AI pipeline data processing tasks to SmartNICs and sketches designs for normalization, bilinear interpolation, and tokenization, without implementing them.

  23. Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference

    cs.AR 2024-12 conditional novelty 5.0 of 10

    Retrieval-augmented generation adds significant latency, with retrieval making up roughly one-third of time-to-first-token, and the retrieval database's memory and throughput scale poorly to production sizes.

  24. MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems

    cs.AI 2025-08 reject novelty 4.0 of 10

    The authors claim their MultiFluxAI orchestration framework achieves 95% accuracy and 0-10 ms responses by combining rule-based routing, caching, and graph knowledge stores for multi-service RAG queries.

  25. Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Caching query-aware LLM document summaries and reusing them for similar questions cuts input token use by roughly half while keeping answer similarity close to full-document processing.

  26. EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration

    cs.DC 2025-04 conditional novelty 4.0 of 10

    A time-sliced, cyclically staggered scheduling strategy, partial disaggregation, is shown to improve SLO-aware goodput for 30B to 72B LLMs on commodity-interconnect clusters relative to co-located and fully disaggrega...

  27. Adaptive Contextual Caching for Mobile Edge Large Language Model Service

    cs.NI 2025-01 reject novelty 4.0 of 10

    An edge cache for RAG-driven LLMs learns its replacement policy with deep reinforcement learning and claims over 80% hit rates after 11 training episodes.

  28. Parallel Key-Value Cache Fusion for Position Invariant RAG

    cs.AI 2025-01 conditional novelty 4.0 of 10

    KV-Fusion encodes each retrieved passage in parallel with identical local positions, fuses the key-value caches, and fine-tunes a decoder so answers do not depend on passage order.

  29. A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A systematic review of retrieval-augmented generation that organizes progress by year and application but introduces no new measurements or results.

  30. A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models

    cs.IR 2025-07 reject novelty 3.0 of 10

    A survey claims proactive defenses against LLM misinformation outperform post-hoc detection by up to 63%, but no meta-analysis details are provided to support the claim.

  31. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

  32. Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers

    cs.IR 2025-05 conditional novelty 2.0 of 10

    A survey of retrieval-augmented generation methods, organized by architecture, with a comparative analysis of published QA results.

Pith tools