Predictable LLM Serving on GPU Clusters

Erfan Darzi; Shreeanant Bharadwaj; Sree Bhargavi Balija

REVIEW 4 major objections 5 minor 12 references

Predictable LLM Serving on GPU Clusters

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A controller that runs inside a tenant VM cuts GPU SLO violations by 32 percent on shared A100 clusters, with no fabric privileges.

desk verdict Solid integrated controller with a believable ablation, but the evaluation never accounts for MIG reconfiguration downtime, which could inflate the headline SLO gains. read the letter →

arxiv 2508.20274 v1 pith:A4NOSSWT submitted 2025-08-27 cs.DC

Erfan Darzi , Shreeanant Bharadwaj , Sree Bhargavi Balija This is my paper · ORCID

classification cs.DC

keywords GPUmulti-tenancyLLMservingvTTFTclusterschedulingA100SLOcomplianceQoS

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a cloud tenant can tame noisy-neighbor interference on shared A100 GPUs without any help from the cluster scheduler or fabric privileges. Its host-level controller watches per-tenant tail latencies and PCIe activity, then takes one of three actions: resize the tenant's MIG partition for stronger hardware isolation, move the tenant to a less contended PCIe path, or throttle the offending background workload. In a controlled 2-node, 16-GPU cluster, the controller lowers the SLO miss-rate by about 32 percent and p99 latency by about 15 percent while keeping throughput loss at or below 5 percent relative to static MIG with naive placement. For LLM serving with vLLM, time-to-first-token p99 improves by roughly 10–15 percent at the same cost. The point is that predictable serving does not require privileged access to the GPU fabric.

What carries the argument

The central object is a three-tiered decision space: dynamic MIG reconfiguration, PCIe-aware placement, and lightweight guardrails. The argument is carried by a feedback loop with hysteresis—p99 exceeding threshold for Y consecutive windows triggers escalation, while sustained stability triggers relaxation—combined with the processor-sharing model of PCIe contention, which justifies why throttling background I/O and moving off hot PCIe paths reduce latency tails. The controller's greedy upgrade policy is guaranteed to terminate because each isolation upgrade strictly increases SM/memory resources and the configuration space is finite.

What would settle it

On a cloud platform where the guest cannot reconfigure MIG profiles or read PCIe counters (for instance, a managed GPU service that blocks nvidia-smi mig), run the same controller under the same T2/T3 interference schedule; if the SLO miss-rate remains at the static-MIG baseline, the central claim is falsified. The positive version is a reproduction on a different 16-GPU A100 cluster with the same controller and interference toggles, showing the same ≈32% miss-rate reduction at ≤5% throughput cost.

Watch

Extended reading notes

Core claim

The central claim is that PCIe-fabric interference, not just compute and memory contention, drives tail-latency violations in multi-tenant GPU serving, and that a tenant-visible controller can mitigate it using only three conservative levers: dynamic MIG reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas and cgroup I/O throttles). The paper reports that on one host and on a 2-node (16-GPU) cluster, the full controller reduces SLO miss-rate by approximately 32 percent (about 1.5×) and p99 latency by approximately 15 percent with at most 5 percent throughput cost versus static MIG and naive placement. Ablations show the MIG and placement components contribute nearly equally, with guardrails adding a smaller increment. The same controller, unchanged, improves LLM TTFT p99 by about 10–15 percent at under 5 percent throughput cost when serving OLMo 2 7B Instruct through vLLM.

Load-bearing premise

The controller only works if the cloud lets the tenant VM dynamically change MIG profiles and read PCIe/DCGM counters from inside the guest; on platforms that hide these capabilities, none of the three action levers can fire.

Editorial extensions

If this is right

Dynamic MIG reconfiguration alone lowers p99 from 20.0 ms to 17.2 ms, PCIe-aware placement alone to 17.8 ms, and the two together to 16.5 ms, so the pair is nearly additive.
The controller's actions are gated by dwell and cool-down timers and validated before persisting, so it can run as a host-level refinement beneath cluster schedulers without causing thrash.
Because the controller needs only root access inside the tenant VM, it can be deployed on rented cloud instances without cooperation from the provider.
For LLM serving, the same controller improves TTFT p99 by roughly 10–15 percent and keeps throughput loss under 5 percent, indicating the mechanism transfers from fixed-size inference to autoregressive generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the controller is deployed on a GPU generation that exposes MIG and guest-visible PCIe counters, the same 32 percent reduction is likely to reproduce, but the paper only demonstrates it on A100; a natural next experiment is an H100 cluster with the same interference schedule.
Because the paper's interference is scripted (T2/T3 toggles), the benefit under uncoordinated production traffic may be smaller; a testable extension is to replay a real datacenter trace of PCIe and SM load.
The controller's placement heuristic is a simple penalty score; replacing it with a learned predictor could improve stability, but would add complexity and require richer signals than NVML/DCGM expose.
The paper's limitation section states that very heavy training tenants will overwhelm the controller, so the practical scope is moderate background noise rather than full training storms.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The punchline: this is a legitimate, well-scoped systems contribution — combining dynamic MIG, PCIe-aware placement, and guardrails into a host-level controller that a tenant can run with root inside a VM — and the empirical story is mostly believable. But there is a hole in the evaluation: the paper never says how requests that arrive during MIG reconfiguration's brief tenant pause are handled, and since dynamic MIG is the biggest single contributor in the ablation, that omission matters for the headline 32% SLO miss-rate reduction.

What is actually new: MIGRator, PCIe-aware scheduling, and host-level throttles all exist in the cited literature. The contribution is the integration into one conservative controller demonstrated without fabric privileges on a 2-node/16-GPU cluster. That's a real, useful thing to show. The ablations are clean and internally consistent — the 20.0→16.5ms p99 progression matches the miss-rate numbers, 7 repeats with CIs is decent, and the vLLM/OLMo TTFT case study strengthens the story.

Soft spots, in order. First, the reconfiguration downtime accounting. Algorithm 1 relaunches after every isolation change; Table 4 says MIG changes take 18±6s, <5/hour. The discussion admits a "brief pause of the tenant." But Section 3.2 says they record the "full latency distribution" without stating whether requests during the pause are queued, failed, timed out, or dropped from the sample. If dropped, the reported p99 and miss-rate are computed on a censored distribution and the gains are overstated. This needs to be explicit. Second, no code or data released — for a systems paper with this many parameters, that hurts reproducibility, especially the placement score heuristic. Third, the baseline is underspecified: what static MIG profile, what "naive" placement? Fourth, the formal model in Section 2.5 is qualitative and the stability theorem is a standard PS argument; it doesn't derive the empirical gains, so it's fine to keep but shouldn't be the centerpiece.

The citation pattern looks fair — the prior work is credited, and the novelty claim is appropriately narrow. The paper does admit its small scale and synthetic workloads in Section 5. In short, the approach is credible and worth engaging, but the downtime accounting must be fixed before the headline numbers mean what they claim.

I'd send this to a serious referee. For a reading group, it's a good case study in how SLO evaluation can quietly miss reconfiguration cost.

Referee Report

4 major / 5 minor

Summary. The paper presents a host-level controller for latency-sensitive LLM inference on shared A100 GPU clusters. The controller combines dynamic MIG reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas, cgroup I/O throttles), driven by per-tenant tail-latency signals and guided by a three-tiered decision policy with dwell and cool-down periods to prevent oscillation. The authors report that on a single host and a 2-node 16-GPU AWS p4d cluster, compared to static MIG and naive placement, the controller reduces SLO miss-rate by about 32%, improves p99 latency by about 15%, and incurs at most 5% throughput cost. A vLLM/OLMo 2 7B case study reports TTFT p99 improvement of about 13% at about 4% throughput cost. The paper also sketches a formal model of PCIe contention as a processor-sharing queue and states a stability theorem for throttled tenants.

Significance. The problem addressed is relevant: noisy-neighbor interference on shared PCIe is a recognized cause of tail latency in multi-tenant GPU inference, and a renter-deployable controller that does not require fabric privileges is a practical contribution. The empirical methodology is generally sound: 7 repeated runs with fixed seeds and 95% confidence intervals, and the reported 32% miss-rate reduction is arithmetically consistent with Table 3 (16.4% to 11.1%). The ablation study is a useful strength, showing that dynamic MIG and placement each contribute comparably. However, the formal model in Section 2.5 is explicitly qualitative, the stability theorem is only a sketch, and there are load-bearing omissions in the experimental reporting (measurement of downtime, multi-node numbers, sensitivity analysis) that currently prevent the claims from being fully supported. The approach's significance is moderate: the gains are modest but consistent, and the deployment story is clear.

major comments (4)

[Sections 3.2, 5, Algorithm 1] The manuscript does not state how requests arriving during MIG reconfiguration or tenant pause are treated in the latency and SLO metrics. Algorithm 1 calls Relaunch(C) after every isolation change, Table 4 reports MIG reconfiguration at 18±6 s and fewer than 5 moves per hour, and Section 5 admits that 'MIG profile changes require a brief pause of the tenant and may reload model state.' This implies the primary tenant may be unavailable for up to roughly 90 s per hour. If requests that arrive during this pause are omitted from the sample, or if the load generator is paused, then the reported p99 and SLO miss-rate are computed on a censored distribution, and the headline 32% miss-rate reduction and 15% p99 improvement are inflated by exactly the downtime the paper acknowledges. The paper must specify explicitly whether such requests are queued, failed, timed out, or excluded, and if excluded, must recompute the metrics with the downtime counted as SLO violations or provide a sensitivity bound.
[Abstract and Section 4] The abstract claims results on 'a single host and a 2-node (16-GPU) cluster,' but no quantitative results for the multi-node case are reported in the Results section. The text states only that 'the policy shows similar improvements' on the 2-node cluster, with no table or figure giving SLO miss-rate, p99, throughput, or confidence intervals for that configuration. Table 3 appears to report only single-host ablation results. The multi-node claim in the abstract and contributions is therefore unsupported by the presented evidence. Please provide the multi-node measurements or revise the claims to match the data actually shown.
[Section 3.3.3 (E3) and Section 4] The paper lists a sensitivity analysis (E3) as one of the experiments, but the Results section contains no sensitivity results: no tables or figures for variations of tau, Y, MPS quota bounds, or I/O throttle bounds. Given that these parameters are free and are central to the controller's behavior, the absence of any reported sensitivity data weakens the robustness claim and makes it impossible to judge whether the reported improvements depend critically on hand-tuned thresholds. Either add the sensitivity results or remove the E3 experiment listing from the experimental design.
[Section 2.5.3, Claim 1] The 'stability theorem' is presented as a formal claim, but the proof is only a sketch. The assumptions are strong (the entire PCIe fabric is a single processor-sharing server, aggregate throttles are always below capacity) and are not verified against the experimental system. However, the paper explicitly labels the model as 'qualitative guidance rather than an exact tail predictor,' so this is not a fatal flaw. Nevertheless, the theorem label overstates the result; I recommend either proving the claim rigorously under assumptions that are checked in the evaluation, or rephrasing it as a heuristic stability argument.

minor comments (5)

[Section 2.2.1] The placement score is described qualitatively ('the score penalizes...'); no concrete formula or weighting is given. For reproducibility, please define the score explicitly as an equation or pseudocode.
[Table 1 and Section 2.3] Dwell time (256 observations) and cool-down (128 observations) are given in observation counts, while Section 2.3 discusses them as time intervals. Since the sampling interval Delta is 1–5 s, please clarify the relationship (e.g., dwell time = 256 * Delta seconds) and state the value used in the experiments.
[Section 3.1] The interference script is said to 'toggle' T2 and T3, but the duty cycle, on/off durations, and phase of the toggling are not specified. Since all comparisons rely on identical interference schedules, please define the schedule precisely.
[Algorithm 1] The function Relaunch(C) is never defined in the algorithm or implementation notes. It presumably restarts the serving process and possibly reloads the model; please state exactly what Relaunch does and how long it takes, since it is directly relevant to the downtime concern.
[Figures 3 and 4] The figures are referenced but are not included in the submitted text; please ensure the actual figures are present and that captions describe the data and, where appropriate, confidence intervals or error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: reported gains are empirical and the formal model is explicitly qualitative.

full rationale

The paper's central claims are empirical. Table 3 reports measured SLO miss-rate and p99 latency across configurations, and the improvements (32% miss-rate reduction, ~15% p99 gain) are direct experimental comparisons against a static-MIG baseline under identical interference schedules. The formal model in Section 2.5 is explicitly used only for qualitative guidance: the paper states 'We report empirical p99/p999 in evaluation and use the bound qualitatively to explain how saturation inflates tails' and 'the model serves as qualitative guidance rather than an exact tail predictor.' No parameter is fitted to the reported improvements; the controller thresholds (tau=15 ms, Y=3, dwell/cool-down, MPS/IO bounds) are stated as configuration choices, not fitted to the outcome. The stability Claim 1 is a standard processor-sharing queue result and does not by itself imply any particular p99 or miss-rate number. References to prior dynamic-MIG and PCIe-scheduling work motivate design choices, but the ablation study independently measures each component's contribution, so no load-bearing claim reduces to a self-citation chain. The acknowledged limitations — brief tenant pauses during MIG reconfiguration, opaque PCIe topology, small-scale synthetic evaluation — are validity/correctness risks rather than circularity, and the specific concern that reconfiguration downtime may censor the latency distribution is a measurement-threat issue, not a case of the derivation being equivalent to its inputs. The paper does not derive its headline numbers from the model, and no equation or fitted parameter is renamed as a prediction; the derivation chain, such as it is, is therefore self-contained and non-circular.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The empirical results do not depend on a derived quantity; the formal model provides qualitative guidance and a standard stability theorem. The main added 'apparatus' is the controller itself, which is a software artifact, not an invented physical entity.

free parameters (7)

Tail threshold (tau) = 15 ms
Hand-set SLO trigger for p99 latency; chosen by the authors, not fit to data.
Persistence (Y) = 3 windows
Number of consecutive windows before triggering; chosen by hand.
Dwell time = 256 observations
Minimum time between policy changes; fixed by configuration.
Cool-down period = 128 observations
Grace period after returning to performance mode; fixed.
MPS quota bounds = 50-100 percent
Bounds on MPS active thread percentage; selected by hand.
I/O throttle bounds = 100-500 MB/s
Bounds on cgroup io.max; selected by hand.
Sampling interval Delta = 1-5 s
Polling interval for latency and system signals; configurable.

assumptions (5)

domain assumption PCIe fabric behaves as a single processor-sharing server of capacity B (Section 2.5.1).
Used to derive qualitative latency guidance and the stability theorem; real PCIe topology is hierarchical with root complexes and switches.
domain assumption Aggregate throttle demands satisfy sum_j g_j < B (Claim 1 condition iii).
Assumed for the stability theorem; the controller is said to never violate this, but the claim is not proven.
domain assumption Arrivals to the latency-sensitive tenant are stationary with lambda_i < mu_i(B) (Claim 1 condition iv).
Needed for finite moments of the latency distribution; not validated in experiments.
standard math Standard queueing results: Kingman approximation and PS queue stability (Section 2.5).
Used to qualitatively justify tail behavior; standard textbook results.
domain assumption A100 MIG supports dynamic reconfiguration with overhead <=30 s, and NVML/DCGM counters are readable from inside the VM (Section 2.4).
Load-bearing for the controller's ability to act; only demonstrated on AWS p4d.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Predictable LLM Serving on GPU Clusters." pith.science (2026). https://pith.science/paper/A4NOSSWT

@misc{pith2026250820274,
  author       = {Pith},
  title        = {Pith review of: Predictable LLM Serving on GPU Clusters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4NOSSWT}},
  note         = {Machine review of arXiv:2508.20274}
}

abstract

Latency-sensitive inference on shared A100 clusters often suffers noisy-neighbor interference on the PCIe fabric, inflating tail latency and SLO violations. We present a fabric-agnostic, VM-deployable host-level controller that combines dynamic Multi-Instance GPU (MIG) reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas, cgroup I/O). It samples per-tenant tails and system signals, uses topology hints to avoid PCIe hot spots, and gates actions with dwell/cool-down to avoid thrash. On a single host and a 2-node (16-GPU) cluster, SLO miss-rate is reduced by \(\approx\)32\% (\(\approx\)1.5) and p99 latency improves \(\approx\)15\% with \(\leq\)5\% throughput cost versus static MIG and naive placement; ablations show MIG and placement contribute comparably. We also evaluate LLM serving with vLLM on OLMo 2 7B Instruct: TTFT p99 improves \(\approx\)10--15\% at \(\leq\)5\% cost without changing the controller.

Figures

Figures reproduced from arXiv: 2508.20274 by the authors.

**Figure 1.** System architecture of the multi-tenancy controller. The monitoring domain detects SLO violations and performs [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** PCIe contention model and topology. Under a PS model with capacity [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Adaptive controller behavior and efficiency. (a) The controller responds to dynamic interference bursts with progres [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The impact of PCIe contention on the tail of the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 8 canonical work pages

[1]

Franklin, Joseph E

Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gon- zalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implemen- tation (NSDI 17). USENIX Association, Boston, MA, 613–627

work page 2017
[2]

Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wen- cong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Predictable LLM Serving on GPU Clusters Clusters for DNN Training Workloads. In Proceedings of the 2019 USENIX An- nual Technical Conference (USENIX ATC ’19) . USENIX Association, 947–960. https://www.usenix.or...

work page 2019
[3]

Baolin Li, Tirthak Patel, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari
[4]

Chen Li, Yifan Sun, Lingling Jin, Lingjie Xu, Zheng Cao, Pengfei Fan, David Kaeli, Sheng Ma, Yang Guo, and Jun Yang. 2019. Priority-Based PCIe Scheduling for Multi-Tenant Multi-GPU Systems. IEEE Computer Architecture Letters 18, 2 (2019), 157–160. https://doi.org/10.1109/LCA.2019.2955119

work page arXiv 2019
[5]

Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkatara- man, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. 2020. Themis: Fair and Efficient GPU Cluster Scheduling. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20) . USENIX Association, 289–304

work page 2020
[6]

Kamran Razavi, Saeid Ghafouri, Max M"uhlh"auser, Pooyan Jamshidi, and Lin Wang. 2024. Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling. In Proceedings of the 4th Workshop on Machine Learning and Sys- tems (EuroMLSys 2024), co-located with EuroSys 2024 . Association for Computing Machinery, 184–191. https://doi.org/10.1145/364297...

arXiv 2024
[7]

Yan-Mei Tang, Wei-Fang Sun, Hsu-Tzu Ting, Ming-Hung Chen, I-Hsin Chung, and Jerry Chou. 2025. PCIe Bandwidth-Aware Scheduling for Multi-Instance GPUs. In Proceedings of the International Conference on High Performance Comput- ing in Asia-Pacific Region (HPC Asia ’25) . Association for Computing Machinery, Hsinchu, Taiwan. https://doi.org/10.1145/3712031.3712324

work page arXiv 2025
[8]

Tianyu Wang, Sheng Li, Bingyao Li, Yue Dai, Ao Li, Geng Yuan, Yufei Ding, Youtao Zhang, and Xulong Tang. 2024. Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration. arXiv preprint arXiv:2407.13126 (2024)

arXiv 2024

Show all 12 references

[9]

Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Trans- parent GPU Sharing in Container Clouds for Deep Learning Workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . USENIX Association, 69–85

2023
[10]

Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018. Gandiva: Introspective Cluster Schedul- ing for Deep Learning. InProceedings of the 13th USENI...

2018
[11]

Shulai Zhang, Ao Xu, Quan Chen, Han Zhao, Weihao Cui, Zhen Wang, Yan Li, Limin Xiao, and Minyi Guo. 2025. Efficient Performance-Aware GPU Sharing with Compatibility and Isolation through Kernel Space Interception. In Proceedings of the 2025 USENIX Annual Technical Conference (...

2025
[2022]

In Proceedings of the 13th ACM Symposium on Cloud Computing (SoCC ’22)

MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters. In Proceedings of the 13th ACM Symposium on Cloud Computing (SoCC ’22). Association for Computing Machinery, 173–189. https://doi.org/10.1145/ 3542929.3563510

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Franklin, Joseph E

Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gon- zalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implemen- tation (NSDI 17). USENIX Association, Boston, MA, 613–627

work page 2017

[2] [2]

Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wen- cong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Predictable LLM Serving on GPU Clusters Clusters for DNN Training Workloads. In Proceedings of the 2019 USENIX An- nual Technical Conference (USENIX ATC ’19) . USENIX Association, 947–960. https://www.usenix.or...

work page 2019

[3] [3]

Baolin Li, Tirthak Patel, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari

[4] [4]

Chen Li, Yifan Sun, Lingling Jin, Lingjie Xu, Zheng Cao, Pengfei Fan, David Kaeli, Sheng Ma, Yang Guo, and Jun Yang. 2019. Priority-Based PCIe Scheduling for Multi-Tenant Multi-GPU Systems. IEEE Computer Architecture Letters 18, 2 (2019), 157–160. https://doi.org/10.1109/LCA.2019.2955119

work page arXiv 2019

[5] [5]

Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkatara- man, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. 2020. Themis: Fair and Efficient GPU Cluster Scheduling. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20) . USENIX Association, 289–304

work page 2020

[6] [6]

Kamran Razavi, Saeid Ghafouri, Max M"uhlh"auser, Pooyan Jamshidi, and Lin Wang. 2024. Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling. In Proceedings of the 4th Workshop on Machine Learning and Sys- tems (EuroMLSys 2024), co-located with EuroSys 2024 . Association for Computing Machinery, 184–191. https://doi.org/10.1145/364297...

arXiv 2024

[7] [7]

Yan-Mei Tang, Wei-Fang Sun, Hsu-Tzu Ting, Ming-Hung Chen, I-Hsin Chung, and Jerry Chou. 2025. PCIe Bandwidth-Aware Scheduling for Multi-Instance GPUs. In Proceedings of the International Conference on High Performance Comput- ing in Asia-Pacific Region (HPC Asia ’25) . Association for Computing Machinery, Hsinchu, Taiwan. https://doi.org/10.1145/3712031.3712324

work page arXiv 2025

[8] [8]

Tianyu Wang, Sheng Li, Bingyao Li, Yue Dai, Ao Li, Geng Yuan, Yufei Ding, Youtao Zhang, and Xulong Tang. 2024. Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration. arXiv preprint arXiv:2407.13126 (2024)

arXiv 2024