Pith. sign in

REVIEW 4 cited by

D\'ej\`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.01876 v1 pith:Y4EOI4VS submitted 2024-03-04 cs.DC

Foteini Strati , Sara Mcallister , Amar Phanishayee , Jakub Tarnawski , Ana Klimovic This is my paper · ORCID
classification cs.DC
keywords efficientavulibbubbleschallengesdeploymentsmemoryproposeserving
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. In this paper, we propose D\'ej\`aVu, a system to address all these challenges using a versatile and efficient KV cache streaming library (D\'ej\`aVuLib). Using D\'ej\`aVuLib, we propose and implement efficient prompt-token disaggregation to reduce pipeline bubbles, microbatch swapping for efficient GPU memory management, and state replication for fault-tolerance. We highlight the efficacy of these solutions on a range of large models across cloud deployments.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Deficit token-budget scheduling plus hybrid prefill overflow makes AR-style colocated and disaggregated serving work for cached diffusion LLMs with block-sized, recurring prefills.

  2. TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

    cs.DC 2026-08 conditional novelty 6.0 of 10

    TensorCast packages tensor lifecycle management into a programmable service layer and reports it can match specialized LLM systems while enabling new cross-component optimization policies.

  3. Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.

  4. Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Select-N meets LLM latency SLOs by choosing and continuously adjusting an offloading interval that controls how much model state lives in CPU memory, increasing host memory usage and throughput versus prior offloading...

Pith tools