Pith. sign in

REVIEW 14 cited by

Preble: Efficient Distributed Prompt Scheduling for LLM Serving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.00023 v2 pith:IOCPZKSV submitted 2024-05-08 cs.DC cs.LG

Vikranth Srivatsa , Zijian He , Reyna Abhyankar , Dongming Li , Yiying Zhang This is my paper
classification cs.DCcs.LG
keywords distributedpreblepromptsschedulingservingllmslatencyprompt
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Prompts to large language models (LLMs) have evolved beyond simple user questions. For LLMs to solve complex problems, today's practices are to include domain-specific instructions, illustration of tool usages, and/or long context such as textbook chapters in prompts. As such, many parts of prompts are repetitive across requests. Recent works propose to cache and reuse KV state of prompts. However, they are all confined to a single-GPU optimization, while production LLM serving systems are distributed by nature. This paper proposes Preble, the first distributed LLM serving platform that targets and optimizes for prompt sharing. We designed a distributed scheduling system that co-optimizes KV state reuse and computation load-balancing with a new scheduling algorithm and a hierarchical scheduling mechanism. Our evaluation of Preble with real workloads and request arrival patterns on two open-source LLMs shows that Preble outperforms the SOTA serving systems by 1.5X to 14.5X on average latency and 2X to 10X on p99 latency.

Discussion (0). Sign in to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

    cs.DC 2026-07 unverdicted novelty 7.0 of 10

    ELDR reduces median TPOT by 5.9-13.9% in PD-disaggregated MoE serving via expert signatures from prefill, K-means partitioning, and locality-band routing with KV-co-indexed signature cache.

  2. Beyond Prediction: Tail-Aware Scheduling for LLM Inference

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Presents a distribution-aware scheduling framework for LLM inference that reduces P99 TTLT by 35-50% and TTFT by 34-47% versus SRPT with perfect length knowledge using statistical signals instead of predictions.

  3. Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference

    cs.DC 2026-05 unverdicted novelty 7.0 of 10

    Kairos improves SLO attainment and throughput in LLM serving by adapting to request length imbalance with priority scheduling and adaptive batching.

  4. Sparse Prefix Caching for Hybrid and Recurrent LLM Serving

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Sparse prefix caching via dynamic programming for optimal checkpoint placement under overlap distributions improves the Pareto frontier for recurrent and hybrid LLM serving on shared-prefix data.

  5. Efficient Remote KV Cache Reuse with GPU-native Video Codec

    cs.DC 2026-02 conditional novelty 7.0 of 10

    KVCodec uses GPU-native video codecs and pipelined fetching to compress and transmit KV caches, delivering up to 3.51x faster TTFT than prior methods while preserving accuracy.

  6. ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

    cs.DC 2026-07 unverdicted novelty 6.0 of 10

    ELDR reduces median TPOT by 5.9-13.9% in PD-disaggregated MoE serving by routing decode requests via prefill-derived expert signatures and K-means locality partitioning over load-balancing baselines.

  7. A Distributed Learned Hash Table

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A learned order-preserving hash inside a Chord-style DHT lets range queries finish in roughly the same cost as single-key lookups, cutting latency and messages by 80-90%+ in tests.

  8. Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving

    cs.DC 2025-07 conditional novelty 6.0 of 10

    Nexus performs proactive intra-GPU disaggregation of prefill and decode, using an analytical cost model and greedy search to dynamically partition SMs, achieving up to 2.2x throughput gains over vLLM.

  9. Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    Omni-Flow introduces a three-layer abstraction (Control Flow, Data Flow, Compute Flow) for unified orchestration and KV cache sharing in multimodal inference pipelines.

  10. Recency/Frequency Adaptive KV Caching for Large Language Model Serving

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    Presents a recency/frequency adaptive KV caching approach that achieves up to 10.8% higher hit rate and 12.6% lower TTFT compared to vLLM on synthetic workloads.

  11. Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    Diffusion Templates is a unified plugin framework that allows injecting various controllable capabilities into diffusion models through a standardized interface.

  12. GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources

    cs.DC 2026-05 unverdicted novelty 4.0 of 10

    GoodServe proposes a predict-and-rectify routing system for agentic LLM inferences on heterogeneous GPUs that improves goodput by up to 27.4%.

  13. Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference

    cs.DC 2026-05 unverdicted novelty 4.0 of 10

    Kairos applies urgency-based priority scheduling on prefill and slack-guided adaptive batching on decode to raise TTFT, TPOT, and end-to-end SLO attainment by up to 33.8% and decode throughput by up to 19.3% versus baselines.

  14. GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving

    cs.NI 2026-02 reject novelty 4.0 of 10

    A cross-region LLM load balancer claims 6.9-15.5% p95 TTFT gains from evolutionary parameter tuning, yet the full text instead reports a 2.5x median gain from hand-tuned weights on single-run benchmarks.

Pith tools