REVIEW 6 cited by
ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Large language model (LLM) serving demands low latency and high throughput, but high load variability makes it challenging to achieve high GPU utilization. In this paper, we identify a synergetic but overlooked opportunity to co-serve latency-critical online requests alongside latency-tolerant offline tasks such as model benchmarking. While promising, existing serving systems fail to co-serve them efficiently, as their coarse-grained resource management at the request or iteration level cannot harvest millisecond-level GPU idle cycles without introducing interference that violates online latency objectives. ConServe is a new LLM co-serving system that achieves high throughput and strong online latency guarantees by managing resources at finer granularities. ConServe introduces three techniques: (1) a latency-aware token-level scheduler that precisely sizes offline batches and tokens to fit within online latency objectives; (2) sub-iteration, layer-wise preemption that allows offline tasks to yield to online load spikes; and (3) incremental KV cache management that enables preempting and resuming offline requests at near-zero cost. Evaluations with Llama-3.1 and Qwen-2.5 models on real-world workloads show that ConServe delivers an average of 2.2$\times$ higher throughput and reduces online serving tail latency by 2.9$\times$ on average compared to state-of-the-art systems.
Forward citations
Cited by 6 Pith papers
-
ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters
A Kubernetes-native scheduler jointly reshapes job configurations, prices cluster resources, and predicts GPU-sharing interference, reporting up to 2.94x lower average JCT and 2.02x higher throughput in testbed and si...
-
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
Cascade coordinates request scheduling and multi-tier KV-cache movement through a single per-request latency budget, improving SLO-satisfied goodput by up to 2.4x in simulation.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
IC-Cache: Efficient Large Language Model Serving via In-context Caching
IC-Cache reuses historical large-model responses as in-context examples so small models can handle a larger share of serving traffic without losing quality, improving throughput and latency.
-
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.
-
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
A MILP scheduler that jointly optimizes GPU composition, deployment configuration, and workload assignment reports 20-41% cost-efficiency gains over homogeneous GPU clusters for LLM serving.
Discussion (0). Continue with ORCID to comment.