REVIEW 13 cited by
NanoFlow: Towards Optimal Large Language Model Serving Throughput
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) have resulted in a surging demand for planet-scale serving systems, where tens of thousands of GPUs continuously serve hundreds of millions of users. Consequently, throughput has emerged as a key metric that determines serving systems' performance. Due to large model sizes and memory-intensive self-attention, LLM serving has been commonly assumed to be memory-bound. Through a detailed analysis, we show that despite having memory-intensive components, end-to-end LLM serving is compute bound for most common workloads and LLMs. Alas, most existing serving engines fall short from optimal compute utilization, because the heterogeneous operations that comprise LLM serving--compute, memory, networking--are executed sequentially within a device. We propose NanoFlow, a novel serving framework that exploits intra-device parallelism, which overlaps the usage of heterogeneous resources within a single device. NanoFlow splits inputs into smaller nano-batches and duplicates operations to operate on each portion independently, enabling overlapping. NanoFlow automatically identifies the number, size, ordering, and GPU resource allocation of nano-batches to minimize the execution time, while considering the interference of concurrent operations. We evaluate NanoFlow's end-to-end serving throughput on several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc. With practical workloads, NanoFlow provides 1.91x throughput boost compared to state-of-the-art serving systems achieving 50% to 72% of optimal throughput across popular models.
Forward citations
Cited by 13 Pith papers
-
Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
Nexus performs proactive intra-GPU disaggregation of prefill and decode, using an analytical cost model and greedy search to dynamically partition SMs, achieving up to 2.2x throughput gains over vLLM.
-
Kinetics: Rethinking Test-Time Scaling Laws
A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.
-
Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
Ascendra improves LLM serving goodput by dynamically reordering prefill requests and offloading near-deadline requests to low-latency instances, achieving up to 1.7x more requests within SLOs than vLLM and Sarathi-Serve.
-
semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
semi-PD is an LLM serving system that disaggregates prefill and decode compute at the streaming multiprocessor level over a unified GPU memory pool, and reports 1.27-2.58x lower average latency and 1.55-1.72x higher S...
-
SYMPHONY: Improving Memory Management for LLM Inference Workloads
SYMPHONY uses advisory requests that signal upcoming multi-turn LLM turns to prefetch KV caches, enabling request-level load balancing and up to 8x more served requests in ShareGPT-based tests.
-
Optimizing ML Concurrent Computation and Communication with GPU DMA Engines
Using DMA engines for collectives (ConCCL) lifts concurrent compute-communication speedup on MI300X from 21% to 72% of ideal.
-
Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
A quantized two-step all-reduce kernel reduces tensor-parallel communication overhead in LLM inference, achieving up to 3.18x faster all-reduce and 2.06x TTFT speedup on L40 GPUs.
-
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.
-
Efficient Clustering with Provable Guardrails for LLM Inference at Scale
Mini-Batch K-Means followed by greedy set-cover within each bucket guarantees every sample lands with a representative that is at least α-similar and attribute-identical, reducing LLM inference cost ~50× at 38M-custom...
-
On Evaluating Performance of LLM Inference Serving Systems
A systematic review identifies eight anti-patterns in LLM inference evaluation and proposes a checklist, with a speculative decoding case study demonstrating how conventional metrics mislead.
-
FinGraV: Methodology for Fine-Grain GPU Power Visibility and Insights
FinGraV reconstructs fine-grain GPU power profiles for sub-millisecond AI kernels by synchronizing CPU-GPU clocks, binning execution times, and separating steady-state execution from steady-state power.
-
EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration
A time-sliced, cyclically staggered scheduling strategy, partial disaggregation, is shown to improve SLO-aware goodput for 30B to 72B LLMs on commodity-interconnect clusters relative to co-located and fully disaggrega...
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
Discussion (0). Continue with ORCID to comment.