REVIEW 16 cited by
Splitwise: Efficient generative LLM inference using phase splitting
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent innovations in generative large language models (LLMs) have made their applications and use-cases ubiquitous. This has led to large-scale deployments of these models, using complex, expensive, and power-hungry AI accelerators, most commonly GPUs. These developments make LLM inference efficiency an important challenge. Based on our extensive characterization, we find that there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Specifically, unlike compute-intensive prompt computation phases, token generation phases do not require the compute capability of the latest GPUs, and can be run with lower power and cost. With Splitwise, we propose splitting the two phases of a LLM inference request on to separate machines. This allows us to use hardware that is well-suited for each phase, and provision resources independently per phase. However, splitting an inference request across machines requires state transfer from the machine running prompt computation over to the machine generating tokens. We implement and optimize this state transfer using the fast back-plane interconnects available in today's GPU clusters. We use the Splitwise technique to design LLM inference clusters using the same or different types of machines for the prompt computation and token generation phases. Our clusters are optimized for three key objectives: throughput, cost, and power. In particular, we show that we can achieve 1.4x higher throughput at 20% lower cost than current designs. Alternatively, we can achieve 2.35x more throughput with the same cost and power budgets.
Forward citations
Cited by 16 Pith papers
-
When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
A simulator predicts that four-way prefill/decode/attention/FFN disaggregation beats unified serving on prefill-heavy agentic workloads by up to 2.06x, but only with stage-specialized custom NPUs and rich enough hardw...
-
Spatial Prefix Caching for Wireless Edge LLM Inference: A Stochastic-Geometry and Queueing Framework
A stochastic-geometry and queueing framework derives prefix-cache association and TTFT coverage for wireless edge LLM inference, revealing load-dependent caching effects.
-
Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units
vNPU virtualizes inter-core connected NPUs via core-ID redirection, range-based memory translation, and topology mapping, achieving up to 1.92x speedup over MIG.
-
DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model
A near-core decompression accelerator plus a 3D roofline model speeds up compressed LLM matrix multiplication by up to 4x in simulation.
-
Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling
AFlex combines operator-level disaggregation with per-operator DVFS to cut LLM serving energy per token by up to 49% without violating P90 TTFT/TPOT SLOs.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Think Before You Grid-Search: Floor-First Triage for LLM Serving
LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.
-
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.
-
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
SwiftSpec uses asynchronous, disaggregated speculative decoding with parallel tree generation and fused kernels to speed up LLM decoding by 1.75x on average over baselines, reaching 348 tokens/s for Llama3-70B on 8 H800 GPUs.
-
Understanding the Performance and Power of LLM Inferencing on Edge Accelerators
An empirical benchmark of a 64GB Jetson Orin AGX shows that LLMs up to 32B parameters can run with INT8 quantization, but token throughput drops sharply as sequence length grows, and quantization slows smaller models.
-
EcoServe: Designing Carbon-Aware AI Inference Systems
EcoServe combines four strategies (reuse, rightsize, reduce, recycle) in an ILP optimizer to cut modeled carbon emissions for LLM serving by up to 47% while keeping SLOs.
-
The Economics of AI Decoding Chips: Rebalancing Compute, Capacity, and Bandwidth for Efficient LLM Inference
A memory-heavy, compute-light decode accelerator can serve a 671B frontier model at roughly one-tenth the capital cost of an eight-GPU node, if the vendor's throughput figures hold.
-
Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications
Overlapping computation and communication in distributed GPU training slows compute kernels by up to 40% and raises power use, while remaining faster than sequential execution.
-
CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
CLONE combines generative pruning, LoRA adapters, a parameter-free Mixture-of-Experts router, and learning-based DVFS to make LLM inference on edge devices faster and more energy-efficient, claiming up to 11.92x speed...
-
GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines
The paper proposes offloading AI pipeline data processing tasks to SmartNICs and sketches designs for normalization, bilinear interpolation, and tokenization, without implementing them.
-
Glinthawk: A Two-Tiered Architecture for Offline LLM Inference
A two-tier inference system that offloads attention and KV cache to cheap CPU nodes raises offline LLM throughput about 6x and lowers hardware cost about 2.8x in a T4-based prototype.
Discussion (0). Continue with ORCID to comment.