Pith. sign in

REVIEW 15 cited by

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.19708 v3 pith:L5WJP2YP submitted 2024-03-23 cs.CL cs.LG

Bin Gao , Zhuomin He , Puru Sharma , Qingxuan Kang , Djordje Jevdjic , Junbo Deng , Xingkun Yang , Zhou Yu
show 1 more author
Pengfei Zuo
This is my paper
classification cs.CLcs.LG
keywords cachescachedattentionconversationsmulti-turnservingaccesscachecomputation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes CachedAttention, a new attention mechanism that enables reuse of KV caches across multi-turn conversations, significantly reducing the repetitive computation overheads. CachedAttention maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache access overheads from slow mediums, CachedAttention employs layer-wise pre-loading and asynchronous saving schemes to overlap the KV cache access with the GPU computation. To ensure that the KV caches to be accessed are placed in the fastest hierarchy, CachedAttention employs scheduler-aware fetching and eviction schemes to consciously place the KV caches in different layers based on the hints from the inference job scheduler. To avoid the invalidation of the saved KV caches incurred by context window overflow, CachedAttention enables the saved KV caches to remain valid via decoupling the positional encoding and effectively truncating the KV caches. Extensive experimental results demonstrate that CachedAttention significantly decreases the time to the first token (TTFT) by up to 87%, improves the prompt prefilling throughput by up to 7.8$\times$ for multi-turn conversations, and reduces the end-to-end inference cost by up to 70%.

Discussion (0). Sign in to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving

    cs.OS 2026-05 unverdicted novelty 7.0 of 10

    Tutti is a GPU-direct SSD-backed KV cache that removes CPU bottlenecks via object abstraction, GPU io_uring, and slack scheduling, delivering near-DRAM performance at 2x higher request rate and 27% lower cost than pri...

  2. Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory

    physics.chem-ph 2026-04 unverdicted novelty 7.0 of 10

    ibDET assembles the full molecular Green's function from multiple small interacting-bath embedding calculations, delivering spectral properties with ~0.1 eV accuracy at far lower cost than full-system methods.

  3. HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

    cs.DC 2026-07 conditional novelty 6.0 of 10

    HyMCache makes SSD-backed CXL-hybrid memory behave like a DRAM-tier KV cache for multi-turn LLM serving by prefetching known prefix blocks and isolating writes, achieving ~3x over local caching and ~70% of distributed...

  4. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  5. KernelSight-LM: A Kernel-Level LLM Inference Simulator

    cs.PF 2026-06 unverdicted novelty 6.0 of 10

    KernelSight-LM simulates token-level LLM inference to predict per-kernel latencies and end-to-end metrics (TTFT, TPOT, throughput) with 12.1% and 3.8% kernel errors in cross-generation and target-measured tiers.

  6. KernelSight-LM: A Kernel-Level LLM Inference Simulator

    cs.PF 2026-06 unverdicted novelty 6.0 of 10

    KernelSight-LM simulates LLM inference at kernel granularity with cross-generation (12.1% per-kernel error) and target-measured (3.8% error) tiers, yielding end-to-end median errors of 15.4%/12.8%/3.0% and 14.3%/6.2%/...

  7. Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    Hive is a multi-agent infrastructure with a logits cache for reducing cross-path redundancy in sampling and agent-aware scheduling for better compute and KV-cache allocation, shown to deliver 1.11x-1.76x speedups and ...

  8. TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing

    cs.DC 2026-04 unverdicted novelty 6.0 of 10

    TokenDance scales multi-agent LLM serving to 2.7x more concurrent agents by collective KV cache reuse and block-sparse diff encoding that achieves 11-17x compression.

  9. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

    cs.OS 2025-11 unverdicted novelty 6.0 of 10

    Continuum applies a time-to-live mechanism to KV cache retention during tool calls in multi-turn LLM agents, reporting over 8x faster average job completion times on benchmarks including SWE-Bench with models up to 35...

  10. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

    cs.OS 2025-11 conditional novelty 6.0 of 10

    TTL-based KV-cache pinning that uses predicted tool-call durations and queueing-delay costs cuts agent job completion time by up to 8x.

  11. Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.

  12. Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

    cs.AR 2026-06 unverdicted novelty 5.0 of 10

    AsymCache combines Multi-Segment Attention, position-aware eviction, and adaptive chunking to cut TTFT by up to 2.03x and TPOT by up to 1.71x versus recent baselines in LLM serving.

  13. Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory

    physics.chem-ph 2026-04 unverdicted novelty 5.0 of 10

    Molecular ibDET assembles full-system GW/EOM-CCSD Green's functions from small atom-centered impurity problems and recovers IPs/EAs within about 0.1 eV of full-system results.

  14. ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories

    cs.DC 2026-06 unverdicted novelty 4.0 of 10

    ITME uses CXL-hybrid memories for byte-addressable remote memory expansion in LLM inference, achieving up to 35.7% throughput improvement over conventional CPU-offloading.

  15. Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stack

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    Workload-aware optimizations for LLM serving in AML and fraud detection yield substantial gains in throughput, latency, and GPU utilization on synthetic compliance prompts.

Pith tools