REVIEW 15 cited by
EPIC: Efficient Position-Independent Caching for Serving Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss.
Forward citations
Cited by 15 Pith papers
-
QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving
QCFuse achieves full-prefill quality in RAG with 1.7x average prefill speedup over full prefill and 1.5x over ProphetKV via compressed query-aware cache fusion.
-
CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference
CachePrune enables fine-grained, token-level KV cache reuse across LLM requests by masking sensitive segments, eliminating direct side-channel leakage while cutting TTFT by 4.5x and raising hit rates by 44% versus pri...
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
ibDET assembles the full molecular Green's function from multiple small interacting-bath embedding calculations, delivering spectral properties with ~0.1 eV accuracy at far lower cost than full-system methods.
-
Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV
Source-omitted event KV rows can carry compact upstream state (semantic materialization); deliberate answer-free carriers raise recovery from 6% to 51% on Qwen3-8B, while passive natural mentions do not.
-
Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
FCGraft synthesizes code policies for embodied agents by grafting KV caches from a library of validated functions, claiming 18.31% higher success rate and 2.3x faster synthesis than prompt-level caching.
-
Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?
GroundedCache reduces unsafe-served rate in RAG answer caching to 0-1.5% (vs 15-51.5% naive) via four validation gates while keeping p50 latency within 1.07x of no-cache baseline.
-
TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
TokenDance scales multi-agent LLM serving to 2.7x more concurrent agents by collective KV cache reuse and block-sparse diff encoding that achieves 11-17x compression.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Selecting tokens for KV recomputation by prompt-attention norms, computed under inference-consistent global RoPE positions, beats CacheBlend and EPIC on long-context QA in LLMs and VLMs.
-
MiniPIC: Flexible Position-Independent Caching in <100LOC
MiniPIC enables multiple position-independent caching methods inside vLLM via unrotated KV storage, per-request RoPE application, and three primitives, delivering 49% prefill throughput gains and up to 100x lower cach...
-
Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
By reusing validated function-level KV caches (stitching) and regenerating only localized error spans (patching), FCGraft makes CodeLLM policies for embodied agents faster and more robust than prompt-level caching.
-
SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance
SIFT precomputes selective attention indices via local and cross-attention invariance to speed RAG prefill 1.71x while keeping accuracy within 1% of full recompute, storing only bit vectors 24,000x smaller than KV tensors.
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
Molecular ibDET assembles full-system GW/EOM-CCSD Green's functions from small atom-centered impurity problems and recovers IPs/EAs within about 0.1 eV of full-system results.
-
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Attention-norm under inference-consistent RoPE identifies tokens that both match the query and can propagate information, enabling better selective KV recomputation and chunk reordering for long-context RAG.
-
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
The paper surveys human memory categories, maps them to LLM memory, and proposes a new three-dimension (object, form, time) categorization into eight quadrants to organize existing work and highlight open problems.
Discussion (0). Sign in to comment.