REVIEW 6 cited by
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation length is generally unpredictable, it is difficult to estimate the time and memory required to process requests, thus posing a challenge for effective request scheduling. Conventional sequence-level scheduling (SLS) serves requests in a first-come first-served (FCFS) manner with static batching where requests with short generation lengths are delayed until those with long ones have finished generation. Besides, to avoid out-of-memory (OOM) errors, SLS batches requests using a small batch size, which limits throughput. Recently proposed iteration-level scheduling (ILS) improves this with continuous batching, timely completing requests and dynamically adding new ones, but often limits the number of parallel-processing requests to OOM errors, thus compromising throughput. Moreover, both SLS and ILS fail to effectively balance workload across multiple LLM instances. To tackle these challenges, we propose slice-level scheduling (SCLS). By splitting the predefined maximal generation length limit into slices and serving batches slice by slice, it provides a precise range of serving time and memory usage for batched requests, laying the foundation for effective scheduling. Experiments confirm that compared with SLS and ILS schedulers, SCLS can improve throughput by up to 315.8% and greatly mitigate load imbalance with proposed batching and offloading algorithms.
Forward citations
Cited by 6 Pith papers
-
Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving
NeuScale routes LLM inference requests to the most energy/cost-efficient configuration of heterogeneous NPU chips using roofline allocation and runtime auto-scaling.
-
WindVE: Collaborative CPU-NPU Vector Embedding
WindVE offloads peak vector-embedding queries to idle CPUs through a regression-calibrated queue manager, achieving up to 22.3% higher concurrency than a GPU/NPU-only baseline under the same SLOs.
-
DeServe: Towards Affordable Offline LLM Inference via Decentralization
DeServe combines KV cache offloading and microbatch scheduling to maintain high LLM offline inference throughput in high-latency decentralized environments.
-
SLO-Aware Scheduling for Large Language Model Inferences
A simulated-annealing scheduler that orders LLM inference requests by their service-level objectives and predicted lengths improves SLO attainment by up to 5x over vLLM and LMDeploy.
-
Multi-Bin Batching for Increasing LLM Inference Throughput
Grouping LLM requests into bins by predicted execution time before forming batches provably raises inference throughput, approaching the theoretical maximum as bin count grows.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
Discussion (0). Continue with ORCID to comment.