Pith. sign in

REVIEW 10 cited by

KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.16525 v2 pith:AB5Q3VUB submitted 2025-03-17 cs.CL cs.AI

Huan Yang , Renji Zhang , Mingzhe Huang , Weijun Wang , Yin Tang , Yuanchun Li , Yunxin Liu , Deyu Zhang This is my paper
classification cs.CLcs.AI
keywords cachekvshareaccuracyreusesystemttftcomparedduring
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances in long-text understanding have pushed the context length of large language models (LLMs) up to one million tokens. It boosts LLMs's accuracy and reasoning capacity but causes exorbitant computational costs and unsatisfactory Time to First Token (TTFT). KV cache reuse, which reuses the exact same KV cache of prefixes and templates or shares similar ones but with extra selective recomputation, offers a promising way to tackle this issue. However, prior studies overlook the cross-request KV reuse and the attention deviations introduced by new tokens during the decoding stage. In this paper, we present a KV cache management module that shares the KV cache across requests under multi-tenant scenarios without sacrificing model accuracy. Our system, KVShare, enables accurate and efficient LLM serving by 1) a Dual-Stage High Deviation algorithm (DHD) that conditionally selects a small portion of KV cache to be recomputed during both prefill and decode phases, and 2) a cache-aware scheduler that prioritizes requests based on their KV cache hit rates and orchestrates continuous batching to achieve enhanced system efficiency and faster TTFT. Multi-task experiments conducted on models such as Qwen2.5-7B,Llama3.1-8B and Yi1.5-9B demonstrate that KVShare reduces TTFT by up to 9.39x and increases 1.2x of the throughput compared to the full KV recompute. Moreover, KVShare achieves 20.38% boost in terms of accuracy compared to SOTA methods.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HijackKV: New Threat in Position-Independent KV Cache Reuse

    cs.CR 2026-07 conditional novelty 7.0 of 10

    A benign text chunk whose KV cache was computed under an adversarial prefix can hijack a victim's LLM output under position-independent cache reuse.

  2. HijackKV: New Threat in Position-Independent KV Cache Reuse

    cs.CR 2026-07 conditional novelty 7.0 of 10

    An attacker can poison a cached chunk's key-value state with a crafted prefix so that a later victim reusing that chunk gets the attacker's chosen answer, with no malicious text in the victim's input.

  3. HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching

    cs.DC 2026-07 accept novelty 7.0 of 10

    Hypic enables position-independent KV/state reuse on hybrid-attention LLMs via cached segment transitions, seam-window repair, and inter-instance segment parallelism, cutting TTFT 3.25x over prefix caching.

  4. Leyline: KV Cache Directives for Agentic Inference

    cs.DC 2026-05 unverdicted novelty 7.0 of 10

    Leyline adds a policy-directed KV cache edit primitive with closed-form RoPE correction for agentic inference, reporting +11.2 pp cache-hit lift and +14.3 pp solve-rate gain.

  5. C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

    cs.CL 2026-07 conditional novelty 6.0 of 10

    C2KV trains a sidecar extractor to produce compressed, position-agnostic KV cache segments that can be concatenated at inference time, reducing prefill, storage, and memory-bandwidth costs without modifying the frozen...

  6. MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference

    cs.AI 2026-07 conditional novelty 6.0 of 10

    MiniCache caches the shared computation of similar LLM requests as parameterized programs, using a small model to fill in changing variables, cutting latency up to 3.1x and raising throughput about 2.8x in benchmarks.

  7. HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching

    cs.DC 2026-07 unverdicted novelty 6.0 of 10

    Hypic enables position-independent KV caching for hybrid-attention models via segment-cumulative operators and boundary seam recomputation, delivering 2.45x average TTFT reduction and up to 2.0x throughput gain.

  8. Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Dual-pool token-budget routing for LLM serving reduces GPU-hours by 31-42% and preemption rates by 5.4x through online-learned request classification without a tokenizer.

  9. OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism

    cs.RO 2026-03 unverdicted novelty 6.0 of 10

    OxyGen unifies KV cache management in MoT VLAs to enable cross-task KV sharing and cross-frame continuous batching, delivering up to 3.7x speedup with 200+ tokens/s language and 70 Hz action on on-device platforms.

  10. SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving

    cs.PF 2026-06 unverdicted novelty 5.0 of 10

    SparseX adds segment-level KV cache reuse with Sparse-Q guided recomputation and layer-wise hybrid attention to handle interleaved serving patterns beyond standard prefix caching.

Pith tools