Pith. sign in

REVIEW 14 cited by

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.02310 v3 pith:GHGZHR4S submitted 2024-03-04 cs.LG cs.DC

Amey Agrawal , Nitin Kedia , Ashish Panwar , Jayashree Mohan , Nipun Kwatra , Bhargav S. Gulavani , Alexey Tumanov , Ramachandran Ramjee This is my paper · ORCID
classification cs.LGcs.DC
keywords sarathi-servelatencydecodeiterationsprefillservingbatchingcapacity
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per request. This makes batching highly effective for decodes and consequently for overall throughput. However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency. We introduce an efficient LLM inference scheduler, Sarathi-Serve, to address this throughput-latency tradeoff. Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency. Furthermore, uniform batches in Sarathi-Serve ameliorate the imbalance between iterations resulting in minimal pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware under tail latency constraints. For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity and up to 3.7x higher serving capacity for the Yi-34B model on two A100 GPUs as compared to vLLM. When used with pipeline parallelism on Falcon-180B, Sarathi-Serve provides up to 5.6x gain in the end-to-end serving capacity. The source code for Sarathi-Serve is available at https://github.com/microsoft/sarathi-serve.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 15 citations worldwide. Full citation record

  1. Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling

    cs.DC 2026-08 conditional novelty 7.0 of 10

    A resource-fair batching policy (ISJL) that keeps co-batched LLM requests within a token-progress window is proved 3/4-competitive in an offline model and empirically outperforms FCFS, SJF, and LJF on throughput and latency.

  2. Spatial Prefix Caching for Wireless Edge LLM Inference: A Stochastic-Geometry and Queueing Framework

    cs.IT 2026-08 accept novelty 7.0 of 10

    A stochastic-geometry and queueing framework derives prefix-cache association and TTFT coverage for wireless edge LLM inference, revealing load-dependent caching effects.

  3. Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

    cs.OS 2026-07 conditional novelty 7.0 of 10

    An ACK-latched epoch protocol lets a hard-barriered CPU LLM inference gang change membership per token bit-exactly and Pareto-dominate static core partitions on general throughput.

  4. Persistent Computational State: A Session-Centric Runtime for Generative World Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    World-model persistence failures are partly a serving-layer bug: restoring the runtime's non-recomputable state (observation+RNG, memory bank, or KV context) after an excursion reproduces the never-left continuation b...

  5. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0 of 10

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

  6. Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

    cs.DC 2025-09 conditional novelty 6.0 of 10

    Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.

  7. Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.

  8. PolyServe: Efficient Multi-SLO Serving at Scale

    cs.DC 2025-07 conditional novelty 6.0 of 10

    PolyServe separates LLM requests into per-token-latency bins, routes to the busiest SLO-attainable server, and lazily fills tight-SLO servers with looser requests, achieving 1.23x simulated goodput.

  9. HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

    cs.DC 2025-02 conditional novelty 6.0 of 10

    A scheduling algorithm for disaggregated LLM inference on heterogeneous GPUs that improves throughput by up to 2x and cuts cost by 30% relative to state-of-the-art serving systems.

  10. EcoServe: Designing Carbon-Aware AI Inference Systems

    cs.DC 2025-02 conditional novelty 6.0 of 10

    EcoServe combines four strategies (reuse, rightsize, reduce, recycle) in an ILP optimizer to cut modeled carbon emissions for LLM serving by up to 47% while keeping SLOs.

  11. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AdaServe customizes speculative decoding per request to satisfy heterogeneous per-token latency SLOs, claiming up to 4.3x fewer violations and 1.9x higher goodput than baselines.

  12. Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization

    cs.DC 2025-02 reject novelty 4.0 of 10

    A hybrid offline-online scheduler claims to raise LLM inference hardware utilization by about 9 points, but its MIP model and 'Lagrangian' rules contain mathematical flaws.

  13. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

  14. Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions

    eess.SY 2025-01 conditional novelty 2.0 of 10

    The paper surveys recent work, models, applications, and challenges of using LLMs in intelligent transportation systems, without presenting new experimental results.

Pith tools