MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs
read the original abstract
New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments, including deep learning training and inference characterization on MIG, GPU sharing characterization, and framework compatibility with MIG. The results of these experiments provide new insights and guidance for users to effectively employ MIG, and lay the foundation for further research on the orchestration of hybrid training and inference workloads on MIGs. The code and results are released on https://github.com/MLSysOps/MIGProfiler. This work is still in progress and more results will be published soon.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs
Festina reduces energy consumption by up to 56% for serverless LLM inference on shared GPUs while keeping TTFT/TBT SLO attainment within 2% of four state-of-the-art baselines.
-
SMART-MIG: A Learning Framework for Scalable and Energy-Efficient GPU Scheduling
SMART-MIG applies MF-MARL for constant-complexity MIG repartitioning plus heuristics for scheduling, reporting 18% better energy-tardiness efficiency than static partitioning and 27% above a theoretical energy lower bound.
-
CompPow: A Case for Component-level GPU Power Management
CompPow makes the case that component-aware power management inside GPUs can yield 10% higher energy efficiency and 5% better performance for ML workloads.
-
A comprehensive evaluation of spatial co-execution on GPUs using MPS and MIG technologies
MPS can boost performance up to 30% and cut energy 20% with careful provisioning but degrades sharply under memory contention, whereas MIG delivers steadier gains through hardware isolation at the cost of higher overh...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.