REVIEW 13 cited by
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes CachedAttention, a new attention mechanism that enables reuse of KV caches across multi-turn conversations, significantly reducing the repetitive computation overheads. CachedAttention maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache access overheads from slow mediums, CachedAttention employs layer-wise pre-loading and asynchronous saving schemes to overlap the KV cache access with the GPU computation. To ensure that the KV caches to be accessed are placed in the fastest hierarchy, CachedAttention employs scheduler-aware fetching and eviction schemes to consciously place the KV caches in different layers based on the hints from the inference job scheduler. To avoid the invalidation of the saved KV caches incurred by context window overflow, CachedAttention enables the saved KV caches to remain valid via decoupling the positional encoding and effectively truncating the KV caches. Extensive experimental results demonstrate that CachedAttention significantly decreases the time to the first token (TTFT) by up to 87%, improves the prompt prefilling throughput by up to 7.8$\times$ for multi-turn conversations, and reduces the end-to-end inference cost by up to 70%.
Forward citations
Cited by 13 Pith papers
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
Molecular ibDET assembles full-system GW/EOM-CCSD Green's functions from small atom-centered impurity problems and recovers IPs/EAs within about 0.1 eV of full-system results.
-
TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure
TensorCast packages tensor lifecycle management into a programmable service layer and reports it can match specialized LLM systems while enabling new cross-component optimization policies.
-
A CXL Memory Rack for Multi-Turn LLM Serving
HyMCache makes SSD-backed CXL-hybrid memory behave like a DRAM-tier KV cache for multi-turn LLM serving by prefetching known prefix blocks and isolating writes, achieving ~3x over local caching and ~70% of distributed...
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
TTL-based KV-cache pinning that uses predicted tool-call durations and queueing-delay costs cuts agent job completion time by up to 8x.
-
Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts
A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.
-
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
A dependency-aware unified caching manager and cost-model swapper reduce Time-To-First-Token for multi-LoRA LLM serving by roughly 50 to 60 percent over vLLM and S-LoRA.
-
Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
Cache-Craft reuses stored attention states for repeated RAG text chunks with selective recomputation, cutting redundant LLM prefill computation by 51% over prefix caching while retaining roughly 90% of answer quality.
-
SYMPHONY: Improving Memory Management for LLM Inference Workloads
SYMPHONY uses advisory requests that signal upcoming multi-turn LLM turns to prefetch KV caches, enabling request-level load balancing and up to 8x more served requests in ShareGPT-based tests.
-
FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving
FastSwitch reduces context-switching overhead in fairness-aware LLM serving with coarse-grained KV cache allocation, asynchronous multithreaded swapping, and KV cache reuse, reporting 1.4-11.2x tail-latency speedups o...
-
InstCache: A Predictive Cache for LLM Serving
A cache that pre-generates answers to instructions a language model judges likely, achieving hit rates up to 2.3x the exact-repetition upper bound on WildChat and up to 50% faster token generation in vLLM.
-
Towards Sustainable Large Language Model Serving
Under light, single-request loads, an older Nvidia T4 GPU can beat a newer RTX 6000 Ada on energy and total carbon for small LLaMA models, but mainly in low-carbon electricity regions.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
Discussion (0). Continue with ORCID to comment.