Pith. sign in

REVIEW 6 major objections 5 minor 33 references

Managing Multi Instance GPUs for High Throughput and Energy Savings

T0 review · 6 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a single A100 GPU's MIG partitions can be dynamically right-sized with memory prediction, yielding up to 6.20x throughput and 5.93x energy savings compared with sequential execution.

desk verdict A real MIG scheduler with a strong Rodinia evaluation, but the linear memory predictor and single-job LLM baseline leave the headline gains under-supported. read the letter →

arxiv 2508.18556 v1 pith:CHYS7TFT submitted 2025-08-25 cs.DC

Abhijeet Saraha , Yuanbo Li , Chris Porter , Santosh Pande This is my paper · ORCID
classification cs.DC
keywords multi-instanceGPUpartitionschedulingmemorypredictiontimeseriesout-of-memoryrecoveryLLMinferenceenergyefficiencyutilization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the concurrency offered by Multi-Instance GPU (MIG) hardware can be systematically exploited by treating partition sizing as a scheduling problem. It combines three memory-estimation routes: compiler analysis for scientific programs, offline model-size estimation for deep networks, and a runtime time-series forecast for workloads whose memory grows during execution, such as LLMs with expanding context windows, and it uses those estimates to place each job on the smallest legal partition. A partition manager models all valid MIG layouts as a finite state machine and chooses placements that preserve the most future configurations, with fusion and fission used to create tight partitions on demand. The measured payoff on an A100 40GB, against a baseline that runs the same batch sequentially on the unpartitioned GPU, is up to 6.20x throughput and 5.93x energy savings for general workloads, 1.59x and 1.12x for ML workloads, and 1.43x and 1.11x for LLM workloads.

What carries the argument

The load-bearing object is the time-series peak-memory predictor: per-iteration instrumentation of the framework's caching allocator, a linear regression $\hat{m}_t = a t + b$ for requested memory, a normal residual model giving a 99% confidence interval, and a separately fitted inverse-reuse-ratio model that converts requested memory into physical memory demand. Its job is to make the early-restart optimization possible. The second mechanism is the partition manager, a finite-state machine over all valid MIG partition states whose offline-precomputed reachability score $\mathrm{fcr}(s)$ counts how many fully configured layouts remain reachable; online allocation picks the placement with the largest score, and fusion/fission operations create tight slices on demand. These two mechanisms together let the scheduler pack jobs tightly without stranding the GPU in a fragmented configuration.

What would settle it

Run a model whose memory footprint is flat for many iterations and then jumps abruptly, for example from 4GB to 9GB at iteration 50, while the predictor is active. If the linear 99% interval computed at iteration 5 projects a peak below the partition size and the job then OOMs at iteration 50, the predictor's central assumption fails for that workload.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that dynamic memory growth, not static memory size, is what prevents MIG partitions from being packed tightly, and that this growth can be forecast early enough to act on it. The system instruments the ML framework's caching allocator to record requested tensor memory and a reuse ratio at every iteration, fits a linear trend $\hat{m}_t = a t + b$ to each, adds a 99% confidence interval from the residuals to catch peaks, and projects the final peak before the job reaches it. When the projection exceeds the current slice, the scheduler restarts the job on a larger partition instead of waiting for an out-of-memory crash; in the paper's LLM example the predictor flags the overflow at iteration 6 versus the actual crash at iteration 94. Combined with compiler-based memory estimates for scientific jobs and offline model-size estimates for deep networks, the same scheduler handles all three workload classes, and the evaluation reports the throughput and energy gains listed in the abstract.

Load-bearing premise

The early-restart optimization rests on the assumption that a dynamic ML job's memory grows at a roughly steady, predictable rate, so a straight-line forecast with a 99% safety margin can see the peak before the job actually runs out of memory.

Editorial extensions

If this is right

  • A single GPU can serve mixed batches of scientific, DNN-training, and LLM jobs concurrently, with job throughput and energy per job improving over running the same batch one job at a time.
  • LLM inference with growing context windows becomes schedulable on small MIG slices: the predictor can flag an imminent out-of-memory condition during the first few percent of iterations, so the job is moved before the wasted work accumulates.
  • Because the reachability scores are precomputed offline, the online partition manager stays cheap regardless of how many layouts the GPU supports, making the scheme portable to other MIG-capable GPUs.
  • Scheduling policy matters: sorting jobs by memory size and scheduling in uniform slices beats FIFO with on-demand reconfiguration for heterogeneous batches, at the cost of fairness within a batch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the linear-growth assumption is the part most likely to give way; a variant that detects stepwise or piecewise growth, for example a new layer or a context-window expansion, would preserve the early-restart benefit for workloads the current 99% interval can miss.
  • Editorial inference: the future-configuration reachability rule is a greedy choice; for a known batch one could precompute an optimal global layout sequence offline and compare it against the greedy online choices, which would show how much flexibility preservation costs.
  • Editorial inference: the reported multipliers compare against sequential execution on an unpartitioned GPU, so they are upper bounds on the concurrency benefit; against a static MIG layout with good packing, the gains would come mainly from dynamic resizing and early restart, not from packing itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper presents MIGM, a scheduler and partition manager for NVIDIA Multi-Instance GPU (MIG) on A100, aiming to maximize throughput and energy efficiency by dynamically right-sizing MIG partitions. Memory requirements are obtained from three sources: compiler analysis (CASE) for scientific workloads, the DNNMem estimator for static ML models, and a runtime time-series predictor for dynamic ML/LLM workloads. The time-series predictor fits a linear trend to per-iteration requested memory and an inverse reuse ratio, adds a 99% confidence interval based on normal residuals, and triggers an early restart when the predicted peak exceeds the current partition. The scheduler offers a reordering policy (scheme A) and an in-order policy (scheme B), with a partition manager that uses a precomputed reachability metric to choose placements preserving future configurations. Evaluation on an A100 40GB reports up to 6.20x throughput and 5.93x energy savings on Rodinia mixes, 1.59x/1.12x for ML training workloads, and 1.43x/1.11x for LLM workloads, against a sequential non-partitioned baseline.

Significance. If the results hold, MIGM addresses a practical problem: modern GPUs provide MIG features but dynamic reconfiguration is hard, and accurate memory estimation enables tighter packing. The paper integrates existing techniques (compiler analysis, DNNMem) with a new component (time-series peak prediction) and a principled partition-placement policy based on future-configuration reachability. The evaluation is broad, covering scientific, CNN/transformer training, and LLM inference, and the reported internal consistency (e.g., throughput and energy trends moving together) is a plus. The main caveat is that the central new mechanism, the time-series predictor, is evaluated on a small set of workloads under a linear-growth assumption, and the convergence criterion is not specified. With additional validation, this would be a useful contribution to the MIG scheduling literature.

major comments (6)
  1. [Section 3.2.3 / Algorithm 1] The convergence check CONVERGE(mem_pred) that gates the early prediction is never defined. Since the early-restart mechanism (Section 5.2.2) is the credited source of LLM gains, the paper must specify what makes a prediction stable (e.g., a threshold on the change in predicted peak over consecutive iterations, a minimum number of fitted points, or a criterion on the coefficient of determination), and should report the number of iterations required for convergence on each workload. Without this, the reader cannot judge whether the predictor risks premature restarts (wasting energy) or late restarts (wasting iterations), which directly affects the claimed 1.43x/1.11x LLM improvements.
  2. [Section 3.2.3, Eq. (1)] The model m_t = a*t + b with normally distributed residuals is assumed without validation for workloads with nonlinear memory growth. LLM inference with growing context windows can exhibit stepwise or superlinear allocation (e.g., KV-cache blocks, allocator segment growth), and the reported Qwen2 example (11.41GB predicted vs 12.23GB actual, Section 5.2.2) shows a 6.7% under-prediction that is not tested at a partition boundary. The paper needs to evaluate the predictor on workloads with nonlinear or stepwise growth and report whether the early restart still fires before the actual OOM point. This is load-bearing because the LLM results rest on this predictor.
  3. [Section 5, Table 2] The LLM experiments are listed as 'homogeneous mixes' but each has batch size 1, meaning the throughput/energy comparison is for a single job with early restart versus a baseline. The paper should clarify the baseline behavior for these jobs (does the baseline run to OOM and then restart, or is it a clean run to completion on a large partition?), and should present per-job turnaround time and energy per job rather than 'throughput' of a one-job batch. As written, the 1.43x throughput claim for LLMs is not a batch-concurrency throughput claim and may overstate the result.
  4. [Section 5.1 / 5.2] All results are from single runs with no error bars or repeated-run statistics, and energy is derived from nvidia-smi polling at 0.1s. For short jobs such as the myocyte kernel in Table 3 (GPU kernel runtime ~2.6ms), 0.1s polling cannot attribute energy accurately, so the energy savings numbers (up to 5.93x) are not robust. The paper should report variance across at least several runs and use a finer-grained energy measurement (e.g., NVML counters or instrumented CUDA events) for short jobs.
  5. [Section 5 baseline] The baseline is a non-partitioned GPU executing the batch sequentially. This is a weak comparator: the authors' own preliminary experiment (Section 2.2) shows that tight partitions alone give 20.6% throughput improvement over the next-largest partition, and a static MIG configuration (e.g., seven 5GB slices) would already capture part of the reported gains for homogeneous small-job mixes. The paper should compare against at least one static MIG configuration to isolate the benefit of dynamic reconfiguration, partition fusion/fission, and prediction. This is central to the contribution claim.
  6. [Section 3.2.1] The paper asserts that 'PyTorch reserved memory' does not directly cause OOM, so that only allocated memory plus CUDA context must be predicted. However, from the CUDA driver's perspective, memory that PyTorch has reserved and cached is still allocated to the process and counts against the MIG slice's memory limit; a subsequent CUDA malloc can fail when the total reserved size already consumes the partition. Since the predictor's target is defined by this exclusion, the paper should empirically validate the claim (e.g., by showing a workload whose reserved memory exceeds the partition but whose active tensors do not, and confirming no OOM occurs). If the claim is wrong, the prediction target undercounts real memory pressure and early restarts will be systematically late.
minor comments (5)
  1. [Section 6] The related work section refers to 'MIGPRO' where the framework is elsewhere called 'MIGM' (e.g., 'MIGPRO focuses first on the hard problem of memory estimation... MIGM's scheduler uses this information'). This naming inconsistency should be fixed.
  2. [Section 5.2.2] There are several typos: 'iste ad of' should be 'instead of', 'Similarily' should be 'Similarly', and 'we can predicts' should be 'we can predict'. A copyedit pass is needed.
  3. [Section 4.1] The paper states the A100 is 'the state of the art MIG used in industry'; given the paper's own abstract mentions Hopper series (H100, H200), this phrasing should be softened to reflect that A100 is a representative MIG-capable GPU.
  4. [Section 3.2.3] The text says 'A lower reuse ratio indicates more reuse', which is confusing because a reuse ratio, as the name implies, should be higher when more reuse occurs. Please define reuse_ratio precisely (e.g., fraction of a new request served from the cached pool) and then align the inverse-ratio formulation with that definition.
  5. [Table 2 / Figure 4] The ML mix table lists 'Heterogeneous' for Ml1-Ml3 but the column header says 'Type'; the caption for Figure 4 does not state the normalization direction for energy plots (whether higher is better). Please add a note to the figure captions and table headers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the time-series memory predictor is fitted to runtime traces and tested against observed peaks; cited compiler/model-size estimators are independent inputs, not renamings of the measured results.

full rationale

The claimed derivation chain is not circular. The time-series memory predictor (Section 3.2.3) fits a linear model m_t = a*t + b to observed PyTorch allocated-memory requests and residual standard deviation, then projects a future peak; this is an empirical extrapolation from runtime traces, not a quantity defined in terms of the scheduler's throughput or energy outcome. Section 5.2.2 validates the prediction directly against independently observed peaks (e.g., Qwen2 forecast 11.41 GB vs. actual 12.23 GB, Llama-3 forecast 16.64 GB vs. actual 16.63 GB). The scheduler uses the prediction only to decide when to resize a partition, and the evaluation compares against a non-predictive OOM-restart policy and a sequential full-GPU baseline, so no fitted parameter is renamed as a performance result. The Rodinia results rely on the compiler analysis of [4] and the ML results on DNNMem [7] for starting partition sizes; these are independently published tools, not quantities equivalent to the claimed throughput/energy gains. Although [4] shares two authors with this paper, it is not invoked as a uniqueness theorem and does not predetermine the measured improvements. The unspecified CONVERGE check in Algorithm 1 and the linear-growth assumption are correctness/completeness risks for workloads with nonlinear memory growth, but they do not make the prediction equal to its inputs by construction. No equation in the paper reduces a claimed result to a fit or to a self-citation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger shows that the framework depends on the hardware-specific MIG state model, the linear-growth memory assumption for dynamic ML workloads, and only weakly on external tools. The linear regression coefficients are fitted per workload, making them the main free parameters, while the convergence rule and confidence level are left underspecified.

free parameters (3)
  • linear regression coefficients a and b for requested memory trend = per workload, not reported
    Section 3.2.3: \m\t = a*t + b is fit online to observed memory requests; the predicted peak is a*t + b + z*sigma. These coefficients are the core of the early OOM prediction.
  • z-score for 99% confidence interval = not stated explicitly, standard z around 2.576
    Used to inflate the predicted peak by z*sigma; chosen to define 'peak' but the 99% level is arbitrary and not tied to OOM risk.
  • convergence threshold for early prediction stop = not specified
    Algorithm 1 checks 'CONVERGE(mem_pred)' but no definition is given; the stopping iteration determines prediction accuracy and restart timing.
assumptions (5)
  • domain assumption A100 MIG hardware exposes a fixed, finite set of valid partition configurations and placement constraints.
    Used throughout Section 2.2 and 4.2; taken from NVIDIA MIG user guide [14]; the FSM model requires this finiteness.
  • domain assumption ML/LLM memory usage grows approximately linearly with iterations.
    Section 3.2.3 assumes linear trend for requested memory and inverse reuse ratio; this is the load-bearing premise for the time series predictor.
  • standard math Residuals of memory observations around the linear fit are normally distributed.
    Section 3.2.3 constructs a 99% CI by assuming normal residuals; no distributional test is provided.
  • domain assumption Restarting a workload on a larger MIG slice is cheaper than checkpointing and restoring.
    Section 6 contrasts with MISO's checkpointing, stating MIGM opts for quick restarts; no timing comparison is given.
  • domain assumption Compiler analysis from CASE [4] and DNNMem [7] accurately estimate memory and compute requirements for scheduled general and DNN workloads.
    Section 4.3 relies on these external tools without re-validating them on the A100; if their estimates are wrong, the tight fit and warp folding decisions degrade.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Managing Multi Instance GPUs for High Throughput and Energy Savings." pith.science (2026). https://pith.science/paper/CHYS7TFT

@misc{pith2026250818556,
  author       = {Pith},
  title        = {Pith review of: Managing Multi Instance GPUs for High Throughput and Energy Savings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHYS7TFT}},
  note         = {Machine review of arXiv:2508.18556}
}
read the original abstract

Modern GPUs such as the Ampere series (A30, A100) as well as the Hopper series (H100, H200) offer performance as well as security isolation features. They also support a good amount of concurrency, but taking advantage of it can be quite challenging due to the complex constraints on partitioning the chip. In this work, we develop partitioning and scheduling schemes for a variety of workloads, ranging from scientific to modern ML workloads, including LLMs. We develop several schemes involving dynamic memory estimation, partition fusion and partition fission. We also support process restart to recover from out-of-memory errors for workloads and early restart as an optimization. This approach yields up to 6.20x throughput and 5.93x energy improvements for general workloads; and we see 1.59x and 1.12x improvement to throughput and energy, respectively, for ML workloads on an A100 GPU. We leverage this technique on LLM workloads and show good improvements, including up to 1.43x throughput improvement and 1.11x energy savings.

Figures

Figures reproduced from arXiv: 2508.18556 by the authors.

Figure 1
Figure 1. High-level flow diagram of the MIGM framework [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Memory structure for machine learning workloads. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A100 Configurations 4.1 A100 Architecture [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Normalized performance results on Rodinia and ML workloads. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 29 canonical work pages

  1. [1]

    Basaran and K

    C. Basaran and K. Kang. Supporting preemptive task executions and memory copies in gpgpus. In 2012 24th Euromicro Conference on Real-Time Systems, 2012

    work page 2012

  2. [2]

    Sheaffer, Sang-Ha Lee, and Kevin Skadron

    Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-Ha Lee, and Kevin Skadron. Rodinia: A benchmark suite for heterogeneous comput- ing. In Proceedings of the 2009 IEEE International Symposium on Workload Characterization, IISWC 2009, October 4-6, 2009, Austin, TX, USA, pages 44–54. IEEE Computer Society, 2009

    work page 2009

  3. [3]

    Sheaffer, Michael Boyer, Lukasz G

    Shuai Che, Jeremy W. Sheaffer, Michael Boyer, Lukasz G. Szafaryn, Liang Wang, and Kevin Skadron. A characterization of the rodinia benchmark suite with comparison to contemporary CMP workloads. In Pro- ceedings of the 2010 IEEE International Symposium on Workload Characterization, IISWC 2010, Atlanta, GA, USA, December 2-4, 2010, pages 1–11. IEEE Computer...

    work page 2010

  4. [4]

    CASE: a compiler-assisted scheduling framework for multi-gpu systems

    Chao Chen, Chris Porter, and Santosh Pande. CASE: a compiler-assisted scheduling framework for multi-gpu systems. In Jaejin Lee, Kunal Agrawal, and Michael F. Spear, editors, PPoPP ’22: 27th ACM SIGPLAN Sym- posium on Principles and Practice of Parallel Program- ming, Seoul, Republic of Korea, April 2 - 6, 2022, pages 17–31. ACM, 2022

    work page 2022

  5. [5]

    Zhao, Yanping Huang, Andrew M

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y . Zhao, Yanpi...

    work page 2024

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany...

    arXiv 2024

  7. [7]

    Estimating GPU memory consumption of deep learning models

    Yanjie Gao, Yu Liu, Hongyu Zhang, Zhengxian Li, Yonghao Zhu, Haoxiang Lin, and Mao Yang. Estimating GPU memory consumption of deep learning models. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmer- mann, editors, ESEC/FSE ’20: 28th ACM Joint Euro- pean Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event...

    work page 2020

  8. [8]

    Characterization and prediction of deep learning workloads in large-scale GPU datacenters

    Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, and Tianwei Zhang. Characterization and prediction of deep learning workloads in large-scale GPU datacenters. In Bronis R. de Supinski, Mary W. Hall, and Todd Gam- blin, editors, International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, N...

    work page 2021

Show all 33 references
  1. [9]

    Gdev: First-class GPU resource manage- ment in the operating system

    Shinpei Kato, Michael McThrow, Carlos Maltzahn, and Scott Brandt. Gdev: First-class GPU resource manage- ment in the operating system. In Preceedings of 2012 USENIX Annual Technical Conference, pages 401–412. USENIX, 2012

    2012

  2. [10]

    MISO: exploiting multi- instance GPU capability on multi-tenant GPU clusters

    Baolin Li, Tirthak Patel, Siddharth Samsi, Vijay Gade- pally, and Devesh Tiwari. MISO: exploiting multi- instance GPU capability on multi-tenant GPU clusters. In Ada Gavrilovska, Deniz Altinbüken, and Carsten Binnig, editors, Proceedings of the 13th Symposium on Cloud Computin...

    2022

  3. [11]

    Clover: Toward sustainable AI with carbon- aware machine learning inference service

    Baolin Li, Siddharth Samsi, Vijay Gadepally, and De- vesh Tiwari. Clover: Toward sustainable AI with carbon- aware machine learning inference service. In Dorian 12 Arnold, Rosa M. Badia, and Kathryn M. Mohror, edi- tors, Proceedings of the International Conference for High Per...

    2023

  4. [12]

    Zico: Efficient GPU memory sharing for concurrent DNN training

    Gangmuk Lim, Jeongseob Ahn, Wencong Xiao, Youngjin Kwon, and Myeongjae Jeon. Zico: Efficient GPU memory sharing for concurrent DNN training. In Irina Calciu and Geoff Kuenning, editors, Proceedings of the 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 20...

    2021

  5. [13]

    Veynu Narasiman, Michael Shebanow, Chang Joo Lee, Rustam Miftakhutdinov, Onur Mutlu, and Yale N. Patt. Improving GPU performance via large warps and two- level warp scheduling. In Carlo Galuzzi, Luigi Carro, Andreas Moshovos, and Milos Prvulovic, editors, 44rd Annual IEEE/ACM ...

    2011

  6. [14]

    Mig user guide

    Nvidia. Mig user guide. https://docs.nvidia.com/ datacenter/tesla/mig-user-guide/index.html

  7. [15]

    Chimera: Collaborative preemption for multitasking on a shared gpu

    Jason Jong Kyu Park, Yongjun Park, and Scott Mahlke. Chimera: Collaborative preemption for multitasking on a shared gpu. In Proceedings of the Twentieth Interna- tional Conference on Architectural Support for Program- ming Languages and Operating Systems, page 593–606. ACM, 2015

    2015

  8. [16]

    Compiler- assisted scheduling for multi-instance gpus

    Chris Porter, Chao Chen, and Santosh Pande. Compiler- assisted scheduling for multi-instance gpus. In Yifan Sun, Daniel Wong, and Hoda Naghibijouybari, editors, GPGPU@PPoPP 2022: Proceedings of the 14th Work- shop on General Purpose Processing Using GPU, Vir- tual Event, Seoul...

    2022

  9. [17]

    Rossbach, Jon Currey, Mark Silberstein, Baishakhi Ray, and Emmett Witchel

    Christopher J. Rossbach, Jon Currey, Mark Silberstein, Baishakhi Ray, and Emmett Witchel. Ptask: Operating system abstractions to manage gpus as compute devices. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, page 233–248. ACM, 2011

    2011

  10. [18]

    A preemption-based runtime to efficiently schedule multi-process applications on heterogeneous clusters with gpus

    Kittisak Sajjapongse, Xiang Wang, and Michela Bec- chi. A preemption-based runtime to efficiently schedule multi-process applications on heterogeneous clusters with gpus. In Proceedings of the 22nd International Sym- posium on High-Performance Parallel and Distributed Computin...

    2013

  11. [19]

    Samuel, Stephen McNally, and John Wynkoop

    Tabitha K. Samuel, Stephen McNally, and John Wynkoop. An analysis of gpu utilization trends on the keeneland initial delivery system. In Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the EXtreme to the Campus and ...

    2012

  12. [20]

    Junkyard computing: Repurposing dis- carded smartphones to minimize carbon

    Jennifer Switzer, Gabriel Marcano, Ryan Kastner, and Pat Pannuto. Junkyard computing: Repurposing dis- carded smartphones to minimize carbon. In Tor M. Aamodt, Natalie D. Enright Jerger, and Michael M. Swift, editors, Proceedings of the 28th ACM Interna- tional Conference on A...

    2023

  13. [21]

    Serving DNN models with multi-instance gpus: A case of the reconfigurable machine scheduling problem

    Cheng Tan, Zhichao Li, Jian Zhang, Yu Cao, Sikai Qi, Zherui Liu, Yibo Zhu, and Chuanxiong Guo. Serving DNN models with multi-instance gpus: A case of the reconfigurable machine scheduling problem. CoRR, abs/2109.11067, 2021

    2021 arXiv

  14. [22]

    Gpupool: A holistic approach to fine-grained GPU sharing in the cloud

    Xiaodan Serina Tan, Pavel Golikov, Nandita Vijayku- mar, and Gennady Pekhimenko. Gpupool: A holistic approach to fine-grained GPU sharing in the cloud. In Andreas Klöckner and José Moreira, editors, Proceed- ings of the International Conference on Parallel Ar- chitectures and ...

    2022

  15. [23]

    Tanasic, I

    I. Tanasic, I. Gelado, J. Cabezas, A. Ramirez, N. Navarro, and M. Valero. Enabling preemptive multiprogramming on gpus. In 2014 ACM/IEEE 41st International Sympo- sium on Computer Architecture (ISCA), 2014

    2014

  16. [24]

    Pcie bandwidth-aware scheduling for multi-instance gpus

    Yan-Mei Tang, Wei-Fang Sun, Hsu-Tzu Ting, Ming- Hung Chen, I-Hsin Chung, and Jerry Chou. Pcie bandwidth-aware scheduling for multi-instance gpus. In Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region, HPC Asia 2025, Hsinchu, Taiwan...

    2025

  17. [25]

    Improving GPU multi-tenancy through dy- namic multi-instance GPU reconfiguration

    Tianyu Wang, Sheng Li, Bingyao Li, Yue Dai, Ao Li, Geng Yuan, Yufei Ding, Youtao Zhang, and Xulong Tang. Improving GPU multi-tenancy through dy- namic multi-instance GPU reconfiguration. CoRR, abs/2407.13126, 2024

    2024 arXiv

  18. [26]

    Mlaas in the wild: Workload analysis and scheduling in large-scale heterogeneous GPU clus- ters

    Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. Mlaas in the wild: Workload analysis and scheduling in large-scale heterogeneous GPU clus- ters. In Amar Phanishayee and Vyas Sekar, editors,19th 13 USENIX Sympos...

    2022

  19. [27]

    Flep: Enabling flexible and efficient preemption on gpus

    Bo Wu, Xu Liu, Xiaobo Zhou, and Changjun Jiang. Flep: Enabling flexible and efficient preemption on gpus. In Proceedings of the Twenty-Second Interna- tional Conference on Architectural Support for Program- ming Languages and Operating Systems, ASPLOS ’17, page 483–496, New Yo...

    2017

  20. [28]

    Gandiva: Introspective cluster scheduling for deep learning

    Wencong Xiao, Romil Bhardwaj, Ramachandran Ram- jee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. Gandiva: Introspective cluster scheduling for deep learning. In Proceedings of the 13th USENIX Con...

  21. [29]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jian- hong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  22. [30]

    Towards GPU utilization prediction for cloud deep learning

    Gingfung Yeung, Damian Borowiec, Adrian Friday, Richard Harper, and Peter Garraghan. Towards GPU utilization prediction for cloud deep learning. In Amar Phanishayee and Ryan Stutsman, editors, 12th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud 2020, July 13-14, 20...

    2020

  23. [31]

    Young, Jason Riedy, Thomas M

    Jeffrey S. Young, Jason Riedy, Thomas M. Conte, Vivek Sarkar, Prasanth Chatarasi, and Sriseshan Srikanth. Ex- perimental insights from the rogues gallery. In 2019 IEEE International Conference on Rebooting Comput- ing (ICRC), pages 1–8, Nov 2019

    2019

  24. [32]

    H. Zhou, G. Tong, and C. Liu. Gpes: a preemptive execution system for gpgpu computing. In 21st IEEE Real-Time and Embedded Technology and Applications Symposium, pages 87–97, 2015. A Appendix A.1 Workload Details There are 7 Rodinia mixes, as shown in Table 1. The first four r...

    2015

  25. [33]

    Table 2: The ML mixes used in the experiments

    The are inference workloads (except in the case of FLAN- T5-train, as indicated). Table 2: The ML mixes used in the experiments. Mix Type Jobs Batch Size Ml1 Heterogeneous 1:0:1:0 14 Ml2 Heterogeneous 1:0:0:0 21 Ml3 Heterogeneous 0:0:1:0 18 FLAN-T5-train Homogeneous flan-t5 4 ...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.