REVIEW 9 cited by
Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This paper introduces Helix, a distributed system for high-throughput, low-latency large language model (LLM) serving in heterogeneous GPU clusters. The key idea behind Helix is to formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs, whose nodes represent GPU instances and edges capture both GPU and network heterogeneity through their capacities. Helix then uses a mixed integer linear programming (MILP) algorithm to discover highly optimized strategies to serve LLMs on heterogeneous GPUs. This approach allows Helix to jointly optimize model placement and request scheduling, two highly entangled tasks in heterogeneous LLM serving. Our evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively, compared to existing approaches. Helix is available at https://github.com/Thesys-lab/Helix-ASPLOS25.
Forward citations
Cited by 9 Pith papers
-
HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.
-
Harmonia: End-to-End RAG Serving Optimization
An end-to-end RAG serving framework that uses component-level batching, resource allocation, and runtime prioritization to improve throughput and reduce SLO violations.
-
gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
gLLM uses a token-throttling scheduler that separately balances prefill and decode token counts across pipeline stages, cutting pipeline bubbles and raising LLM serving throughput by 11-398% over vLLM and SGLang.
-
HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
A scheduling algorithm for disaggregated LLM inference on heterogeneous GPUs that improves throughput by up to 2x and cuts cost by 30% relative to state-of-the-art serving systems.
-
EcoServe: Designing Carbon-Aware AI Inference Systems
EcoServe combines four strategies (reuse, rightsize, reduce, recycle) in an ILP optimizer to cut modeled carbon emissions for LLM serving by up to 47% while keeping SLOs.
-
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
AdaServe customizes speculative decoding per request to satisfy heterogeneous per-token latency SLOs, claiming up to 4.3x fewer violations and 1.9x higher goodput than baselines.
-
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
A MILP scheduler that jointly optimizes GPU composition, deployment configuration, and workload assignment reports 20-41% cost-efficiency gains over homogeneous GPU clusters for LLM serving.
-
DeServe: Towards Affordable Offline LLM Inference via Decentralization
DeServe combines KV cache offloading and microbatch scheduling to maintain high LLM offline inference throughput in high-latency decentralized environments.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
Discussion (0). Continue with ORCID to comment.