Power Stabilization for AI Training Datacenters
read the original abstract
Large Artificial Intelligence (AI) training workloads spanning several tens of thousands of GPUs present unique power management challenges. These arise due to the high variability in power consumption during the training. Given the synchronous nature of these jobs, during every iteration there is a computation-heavy phase, where each GPU works on the local data, and a communication-heavy phase where all the GPUs synchronize on the data. Because compute-heavy phases require much more power than communication phases, large power swings occur. The amplitude of these power swings is ever increasing with the increase in the size of training jobs. An even bigger challenge arises from the frequency spectrum of these power swings which, if harmonized with critical frequencies of utilities, can cause physical damage to the power grid infrastructure. Therefore, to continue scaling AI training workloads safely, we need to stabilize the power of such workloads. This paper introduces the challenge with production data and explores innovative solutions across the stack: software, GPU hardware, and datacenter infrastructure. We present the pros and cons of each of these approaches and finally present a multi-pronged approach to solving the challenge. The proposed solutions are rigorously tested using a combination of real hardware and Microsoft's in-house cloud power simulator, providing critical insights into the efficacy of these interventions under real-world conditions.
This paper has not been read by Pith yet.
Forward citations
Cited by 20 Pith papers
-
Inference as Flexibility: Ramp Management for Transmission-Connected AI Data Centres
Hybrid control of inference batch sizes and BESS reduces BESS energy discharge by 71% and peak power by 51% for a 150 MW TCDC while complying with 10 MW/min ramp limits.
-
A Pre-Dispatch Resonance Safety Criterion for AI Training Clusters
Derives a pre-dispatch resonance safety criterion by inverting two-area swing equations, bounding maximum safe AI cluster size at given iteration periods and showing rescheduling benefits on the IEEE 39-bus system.
-
EasyRider: Mitigating Power Transients in Datacenter-Scale Training Workloads
EasyRider uses passive components plus actively controlled energy storage at the rack level, paired with lifetime-maximizing software, to keep AI training power transients inside grid safety limits without code change...
-
The data heat island effect: quantifying the impact of AI data centers in a warming world
AI data centers raise surrounding land surface temperatures by 2°C on average, potentially affecting over 340 million people via local climate changes.
-
Benchmarking Compound AI Applications for Hardware-Software Co-Design
Introduces a benchmarking suite for compound AI applications to support cross-stack performance, cost, and resource analysis for hardware-software co-design.
-
EnclaveScale: Hardware-Assisted Edge-DP for Secure Data Centre Power Telemetry
EnclaveScale delivers attested edge-DP telemetry for data-center GPU power traces via DCAP and Markov-chain models, reporting 0% post-extraction attacks and 131k samples/s throughput on 32 GCP VMs.
-
Source Side Mitigation of AI Datacenter Power Fluctuations with a Hybrid Energy Storage System and Residual Differentiable Predictive Control
A hybrid energy storage system with residual differentiable predictive control reduces AI datacenter-induced grid frequency deviations by over 80 percent in NPCC 140-bus simulations.
-
Source Side Mitigation of AI Datacenter Power Fluctuations with a Hybrid Energy Storage System and Residual Differentiable Predictive Control
A hybrid energy storage system with rule-based frequency allocation and residual differentiable predictive control reduces AI datacenter-induced generator frequency deviations by more than 80 percent in NPCC 140-bus s...
-
GridPilot: Real-Time Grid-Responsive Control for AI Supercomputers
GridPilot achieves 97.2 ms end-to-end grid response on a 3-GPU V100 testbed (6.9x faster than Nordic FFR requirement) and closes 2.5-5.8 pp cooling overhead via PUE-aware control in European grid replays.
-
From Barrier to Bridge: The Case for AI Data Center/Power Grid Co-Design
AI data centers disrupt grid load diversity with rapid hundreds-of-MW swings, requiring explicit co-design between compute and power systems instead of implicit coexistence.
-
Voltage Ride-Through in Large Loads- A Dual PQ Approach
The paper proposes a dual PQ approach for voltage ride-through in large loads, showing that traditional reactive power compensation is limited by infrastructure constraints and that extreme dips may force disconnection.
-
Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers
Workload composition in AI data centers decouples aggregate power variability from short-horizon ramping through asymmetric queueing where batch jobs fill inference-induced idle capacity.
-
Measurement of Generative AI Workload Power Profiles for Whole-Facility Data Center Infrastructure Planning
High-resolution power profiles for AI workloads on H100 GPUs are measured and scaled to whole-facility energy demand using a bottom-up model, with the dataset made public.
-
Provisioning to Runtime Optimization of a 100 MW-Scale AI Cluster
Reports the first claimed end-to-end power management workflow for a hyper-scale 150 MW AI cluster from pre-deployment planning through dynamic runtime control, with measurements from 83K GPUs.
-
ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training
ScaleAcross Explorer jointly optimizes three design dimensions for scale-across training and reports up to 64.62% speedups over production baselines and 37.59% over prior art in testbed and simulation experiments.
-
Stability Enhancement of Centralized UPS Data Center Systems Under Weak-Grid Conditions
Simulations demonstrate instability of PLL-PI control in UPS rectifiers at SCR=2 and show that power-based control improves damping and stability under weak-grid conditions.
-
Replicating Real-World 23-Hz Oscillations Caused by Large Electronic Loads
A representative feedback system and EMT simulations are used to analyze and reproduce the real-world 23-Hz oscillations from a large electronic load, with results matching fault recorder data.
-
Grid Integration of AI Data Centers: A Critical Review of Energy Storage Solutions
A hierarchical review of energy storage technologies for smoothing the sub-second variable loads of AI data centers on the utility grid.
-
AI Infrastructure Sovereignty
AI sovereignty requires coordinated design of data centers, optical networks, and real-time control systems to operate within energy availability and sustainability constraints.
-
Toward Next-Generation AI Data Centers: Power Delivery Architecture Shifts, Emerging Technologies, and Challenges
The paper reviews three stages of power delivery architecture shifts for AI data centers and identifies three enabling technologies: high-voltage-ratio DC/DC converters, facility-level low-voltage DC distribution, and...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.