REVIEW 7 cited by
Taming the Titans: A Survey of Efficient LLM Inference Serving
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Taming the Titans: A Survey of Efficient LLM Inference Serving
read the original abstract
Large Language Models (LLMs) for Generative AI have achieved remarkable progress, evolving into sophisticated and versatile tools widely adopted across various domains and applications. However, the substantial memory overhead caused by their vast number of parameters, combined with the high computational demands of the attention mechanism, poses significant challenges in achieving low latency and high throughput for LLM inference services. Recent advancements, driven by groundbreaking research, have significantly accelerated progress in this field. This paper provides a comprehensive survey of these methods, covering fundamental instance-level approaches, in-depth cluster-level strategies, emerging scenario directions, and other miscellaneous but important areas. At the instance level, we review model placement, request scheduling, decoding length prediction, storage management, and the disaggregation paradigm. At the cluster level, we explore GPU cluster deployment, multi-instance load balancing, and cloud service solutions. For emerging scenarios, we organize the discussion around specific tasks, modules, and auxiliary methods. To ensure a holistic overview, we also highlight several niche yet critical areas. Finally, we outline potential research directions to further advance the field of LLM inference serving.
Forward citations
Cited by 7 Pith papers
-
SNLP: Layer-Parallel Inference via Structured Newton Corrections
SNLP enables layer-parallel Transformer inference by replacing sequential layer execution with structured Newton corrections and SNLP-aware training regularization, yielding up to 2.3x wall-clock speedup on 0.5B model...
-
CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems
CTA-pipelining reduces 2-layer GEMM latency up to 31.8% vs micro-batching and 29.6% vs Tensor Parallelism on 8-GPU H200/B200 systems by CTA-level cross-GPU pipelining.
-
SNLP: Layer-Parallel Inference via Structured Newton Corrections
SNLP achieves up to 2.58x wall-clock speedup on 0.5B Transformers via architecture-specific Newton corrections (IDN/HCN) that enable layer-parallel inference while preserving perplexity in milder settings.
-
SNLP: Layer-Parallel Inference via Structured Newton Corrections
Structured Newton Layer Parallelism (IDN/HCN) replaces sequential Transformer layers with cheap parallel Newton-style updates, trading finite-iteration bias for up to 2.58× inference speedup.
-
Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers
SNLP reduces encrypted Transformer nonlinear depth from L sequential stages to (L−N)+K, cutting symbolic bootstraps ~2.65× with lower error amplification than sequential inference.
-
Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers
SNLP reduces symbolic FHE bootstraps from 53 to 20 on a 0.5B model with +1.2% PPL degradation and lower polynomial-error amplification than sequential inference.
-
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Presents a recency/frequency adaptive KV caching approach that achieves up to 10.8% higher hit rate and 12.6% lower TTFT compared to vLLM on synthetic workloads.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.