Pith. sign in

REVIEW 4 major objections 5 minor 50 references

PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PowerTrip dynamically picks both which and how many datacenters join each training round, cutting time-to-accuracy by up to 50% versus fixed-participant baselines.

desk verdict A well-motivated, clearly-written paper on dynamic power-aware site selection for geo-distributed ML training, but the headline gains likely rest on an internal inconsistency in the compute-time model, so it needs major revision, not as-is acceptance. read the letter →

arxiv 2507.17904 v1 pith:OWGT6ORL submitted 2025-07-23 cs.DC

Talha Mehboob , Luanzheng Guo , Nathan Tallent , Michael Zink , David Irwin This is my paper · ORCID
classification cs.DC
keywords geo-distributedtrainingpower-constraineddatacentersfederatedlearningsiteselectiontime-to-accuracycommunicationoverheadgreedyheuristicpowerutilizationtraces
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that geo-distributed ML training has to settle two questions at once: which datacenters take part, and how many, and that the right answer changes as power availability and network conditions move. It introduces PowerTrip, a system that ranks sites by a power-to-cost score, starts training with a small core of high-power, low-latency sites, and greedily adds sites only while the projected accuracy gain per unit of round time stays positive. The intended payoff is that a power-constrained training run reaches its target accuracy faster than any fixed participant set, because the system avoids paying the communication tax of a large group in the early rounds. The evaluation reports time-to-accuracy reductions of up to 50% against baselines, including a 44% improvement over the best fixed 70-site policy on EMNIST.

What carries the argument

The load-bearing machinery is a two-term score. A power-to-cost score ranks every site by available power divided by communication delay, with adaptive weights that shift emphasis toward power for small updates and toward latency for large ones. A greedy efficiency score, $\eta_k = \Delta\alpha_k/\tau_k$, then compares candidate site counts: $\Delta\alpha_k$ is the projected accuracy gain and $\tau_k$ is the estimated round time, computed as computation time from mean available power plus communication time from the maximum latency among the top $k$ sites. This turns the combinatorial site-selection problem into a per-round local decision, which is what lets the system scale up only when marginal accuracy per minute is positive.

What would settle it

Re-run the EMNIST 45%-target experiment after replacing PowerTrip's unstated accuracy-gain projection with the actually measured per-round accuracy gain from logs; if the policy then selects roughly 70 sites from the first round and its time-to-accuracy approaches the static policy's 1.16 hours, the claimed advantage comes from the projection rather than the greedy structure.

Watch

Extended reading notes

Core claim

The paper's central claim is that the optimum of the power-communication trade-off is not a fixed number of sites, and that a lightweight runtime greedy policy can track it better than offline static selection. Site throughput is modeled as power-limited, round time as computation (inverse aggregate power) plus communication (max latency over selected sites), so adding sites helps until the slowest new link dominates. PowerTrip's policy starts from a small ranked core, considers candidate counts in a window around the current count, and selects the count that maximizes the ratio of projected accuracy gain to projected round time. On EMNIST with a 45% target, the policy reaches target in 0.65 hours versus 1.16 hours for the static optimum at k=70, and it remains the fastest policy under realistic power traces, varying update sizes, and two additional datasets.

Load-bearing premise

The speedup depends on the unstated formula that projects how much accuracy a larger group of sites will add per round; if that projection is wrong, PowerTrip can choose the wrong number of sites and the reported gains shrink.

Editorial extensions

If this is right

  • Starting with a small, high-power, low-latency core avoids the fixed per-round communication tax of a static 70-site policy, cutting EMNIST time-to-accuracy from 1.16 to 0.65 hours.
  • Under real production power traces, PowerTrip remains the fastest policy, reaching target accuracy roughly 24-27% sooner than the optimal static and power-constrained centralized baselines.
  • Adaptive weighting of power versus communication cost keeps the advantage across model update sizes from 1GB to 2GB, with roughly 23% better time-to-accuracy than static and centralized baselines at 2GB.
  • The same dynamic policy generalizes across tasks: 19% faster than the static optimum on next-character prediction and 32% faster on the speech keyword-spotting workload.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same bottom-up logic likely transfers to incremental learning and fine-tuning, where early rounds dominate total time, so paying a high communication cost at the start is especially wasteful.
  • Because the greedy decision rests on an unwritten accuracy-gain projection, a calibrated empirical model of early-round accuracy gains could make the policy reliable under non-IID data or unstable convergence, where recent-trend extrapolation is noisy.
  • PowerTrip's ranking score could also be used to shed low-power or congested sites mid-training; the paper demonstrates upward scaling but does not evaluate downward scaling under rising power demand.
  • Combined with update compression, which shrinks the per-byte communication cost, the optimal number of sites would likely rise; PowerTrip's adaptive weights anticipate this but the interaction is not quantified.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PowerTrip, a system for geo-distributed ML training that dynamically selects a subset of sites at runtime. PowerTrip ranks sites by an adaptive power-to-cost score, then uses a bottom-up greedy policy to add or remove sites based on a projected accuracy gain per unit of estimated round time. The evaluation uses Flower with 40 GPUs emulating 100 sites, synthetic network profiles, and Google power traces. The headline result is that PowerTrip reaches a target accuracy on EMNIST in 0.65 hours, compared with 1.16 hours for a static policy fixed at k=70 sites, a claimed 44% improvement; comparable but smaller gains are reported on other datasets and with Google power traces.

Significance. If the claimed gains are real, PowerTrip addresses a genuine and timely problem: power-constrained data centers with heterogeneous, time-varying power availability, where the optimal number of geo-distributed training sites is workload- and environment-dependent. The use of real Google power traces, a concrete time-to-accuracy metric, and an end-to-end comparison against a fixed-site baseline are strengths. The paper also makes its design modular and framework-agnostic through Flower. However, the significance is conditional: the central evaluation is simulation-based, and both the round-time model and the accuracy-projection function that drive the greedy policy are not specified or validated, so it is not currently possible to determine whether the reported 44% improvement is a property of PowerTrip or an artifact of the simulation's compute model.

major comments (4)
  1. [Sections 2.2 and 2.3, Eq. (7); Algorithm 1 line 13] The compute model is internally inconsistent and this inconsistency is load-bearing for the headline result. Section 2.2 describes synchronous federated averaging in which each site holds a complete model replica and trains for E local epochs on its own data partition; the round time should then be dominated by the slowest site's local training. Equation (7), however, sets T_round(S) proportional to 1/sum_{k in S} C_k(t), a pooled-throughput model in which adding a site directly reduces the compute term without any data re-partitioning. Algorithm 1 line 13 uses yet a third form, T_comp / mean(P_avail(1:k)). Since Section 5.1 states that simulated wall-clock time is computed from formulas, the 0.65h vs 1.16h comparison in Section 5.3 may describe centralized data parallelism rather than the synchronous federated architecture presented in Section 2.2. The authors should specify the exact round-time formula used in the simulator, justify it against the stated architecture, and re-run the evaluation under a max/straggler compute model, or explicitly state how data is re-partitioned when sites are added.
  2. [Section 3.3, Algorithm 1 line 14] The central decision function, ProjectAccuracyGain, is never written as an equation. The text says only that it incorporates recent accuracy trends, a normalization factor k/k_current, and a logarithmic diminishing-returns term. This function determines which candidate site count is selected at every adjustment interval, so the entire dynamic policy rests on an undisclosed, uncalibrated model. The paper should provide the exact formula, its parameters, and an empirical comparison of projected vs. measured accuracy gains, and should report sensitivity of the time-to-accuracy results to this projection.
  3. [Section 5.1, simulated time methodology] The simulation methodology is not reproducible. The paper states that a single physical GPU runs multiple virtual sites sequentially and that simulated wall-clock time is computed by scaling emulated time by the site's power availability, but it does not give the equation that converts sequential execution on one GPU into the reported parallel round time. It is therefore unclear whether the reported times incorporate the per-site straggler component needed for the synchronous architecture, or whether they follow the pooled-throughput model of Eq. (7). The exact simulator formulas, including how communication time and power scaling are combined, must be reported before the headline numbers can be assessed.
  4. [Section 3.1 and Section 5.3] The adaptive weights w_p and w_c in Eq. (8) are described as 'adjusted based on the model update size', but no formula, threshold, or validation is given. The robustness results in Figure 13 depend on these weights, so the reader cannot tell whether PowerTrip is exploiting a tuned configuration or a principled adaptation rule. The paper should specify how w_p and w_c are computed from D_m and show that the chosen values are not the source of the reported gains.
minor comments (5)
  1. [Section 2.3, Eq. (5)] The notation E[Accuracy Gain per Round] introduces an expectation that is not defined; the paper should clarify whether this is an empirical average or an expectation over randomness in data sampling and site selection.
  2. [Algorithm 1, line 13] The line contains apparent typos: it uses T_comp in the max term where T_comm seems intended, and it refers to mean(P_avail(1:w)) with an undefined variable w instead of k. These should be corrected.
  3. [Section 5.3 and Figure 12] The text says PowerTrip is 24% faster than the optimal static policy with Google traces, while the abstract says 25%; the discrepancy should be reconciled.
  4. [Section 5.1 and Figures 9 and 12] The figure labels 'Random (k=100)' are potentially confusing because the baseline is described in the text as choosing a random subset in each round; the paper should clarify whether the random baseline always uses all 100 sites, a fixed 100-site subset, or a random subset per round.
  5. [Section 5.3, Figure 9] The phrase in Section 5.2 that the 45% target is 'the lowest achievable accuracy among all sites' is unclear; the target selection procedure should be stated precisely, since it affects the time-to-accuracy comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PowerTrip's headline gains come from end-to-end simulated measurements against an empirically tuned static baseline, not from a self-referential fit; the underspecified ProjectAccuracyGain is a reproducibility gap, not a circular reduction.

full rationale

PowerTrip's central claim is evaluated by comparing measured time-to-accuracy across policies in a Flower-based emulator (Section 5.1 through 5.3). The static optimal baseline is obtained by exhaustive end-to-end runs (Section 5.2, Eq. 9), and the 0.65h versus 1.16h result (Figure 9) is an actual simulation measurement, not a value recovered from the same formula used to define the policy. The greedy policy's internal score eta_k = Delta_alpha_k / tau_k (Algorithm 1) uses an unstated ProjectAccuracyGain projection on line 14; while this is an underspecified ad hoc model, the paper does not report that projection itself as a prediction, nor define the evaluation metric in terms of it. The architecture equations (Eqs. 1-4) are standard synchronous federated averaging; the analytic round-time idealization in Eq. (7) is an approximation, and the headline results come from emulated end-to-end runs rather than from plugging values into Eq. (7). The only author self-citation (EcoLearn, [30]) appears in related work and is not load-bearing. The missing equation for ProjectAccuracyGain and the apparent inconsistency between Eq. (7) and Algorithm 1 line 13 are correctness and reproducibility concerns, not circularity, because the comparison against an independently tuned static optimum gives the central claim independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several unstated or underspecified parameters (weights, projection constants, hyperparameters) and on domain assumptions about linear power scaling and simulated timing. No new physical entities are introduced.

free parameters (4)
  • Power-to-cost weights w_p and w_c
    Equation 8 uses weights that 'are adjusted based on the model update size' but no mapping or formula is given; they are effectively tuned to make the heuristic work.
  • Accuracy projection constants
    The ProjectAccuracyGain function (Algorithm 1 line 14) uses a normalization factor and logarithmic diminishing-returns term, but the exact bases, coefficients, and offsets are not specified.
  • Dynamic policy hyperparameters
    Initial site count, patience p, adjustment interval, and candidate window n are left undefined in Algorithm 1 and the text, yet they control the scaling behavior.
  • Fixed-radius baseline threshold = 1000 miles
    The Fixed Radius baseline uses a radius 'we assume based on our communication profiling' in Section 5.3, chosen by the authors rather than from a principled method.
assumptions (4)
  • domain assumption Power availability linearly determines computational throughput (Eq. 6 and Eq. 7)
    The model assumes a site's effective GPUs are min(hardware, power budget) and that compute time scales inversely with that count. Real systems may have nonlinearities, thermal limits, and scheduling overheads.
  • domain assumption Round communication time equals the maximum of per-site delays (Eq. 4)
    Synchronous training is bottlenecked by the slowest site; this is standard but ignores overlap of communication and computation and possible pipelining.
  • ad hoc to paper Projected accuracy gain is computable from recent trends and site count
    The greedy policy relies on a verbal projection model that is never written down, and its accuracy is not validated against measured accuracy curves.
  • domain assumption Simulated wall-clock time accurately represents parallel execution
    Section 5.1 describes sequential execution of virtual sites on fewer GPUs with computed time scaling. This assumes perfect parallelism and zero physical resource contention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training." pith.science (2026). https://pith.science/paper/OWGT6ORL

@misc{pith2026250717904,
  author       = {Pith},
  title        = {Pith review of: PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWGT6ORL}},
  note         = {Machine review of arXiv:2507.17904}
}
read the original abstract

The exponential growth of large-scale AI models has led to computational and power demands that can exceed the capacity of a single data center. This is due to the limited power supplied by regional grids that leads to limited regional computational power. Consequently, distributing training workloads across geographically distributed sites has become essential. However, this approach introduces a significant challenge in the form of communication overhead, creating a fundamental trade-off between the performance gains from accessing greater aggregate power and the performance losses from increased network latency. Although prior work has focused on reducing communication volume or using heuristics for distribution, these methods assume constant homogeneous power supplies and ignore the challenge of heterogeneous power availability between sites. To address the challenge of training large models in power-constrained, geo-distributed environments, we introduce PowerTrip, a system that dynamically selects a subset of sites during runtime to optimize the power-communication trade-off. Specifically, PowerTrip selects sites based on a power-to-cost heuristic, prioritizing those with high power availability and low network latency. PowerTrip employs a dynamic greedy approach and uses the marginal gain in training efficiency, i.e., accuracy improvement per unit of time, to optimize for the number of sites where the performance penalty from network overhead negates the benefit of adding more computational power. Our evaluation, which uses real-world Google power traces to model realistic power capacity constraints, demonstrates that PowerTrip can reduce time-to-accuracy by up to 50% compared to existing baseline policies.

Figures

Figures reproduced from arXiv: 2507.17904 by the authors.

Figure 1
Figure 1. Projected growth of data center power demand [36]. ability of the local grid to reliably supply power to it [5, 32]. In addition to limited power availability, data centers may also have limited compute resources due to GPU shortages within a single region [48]. The limited availability of both power and compute resources can significantly degrade the performance of ML training in the data centers. Power and computa… view at source ↗
Figure 2
Figure 2. The power utilization at different sites in Google’s data center power traces. roughly ten thousand machines [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. provides an overview of PowerTrip’s design, which solves the power-communication trade-off by intelligently and dynami￾cally selecting the optimal subset of available sites at runtime to maximize training performance. PowerTrip is designed to integrate with and expand upon existing distributed learning frameworks. Power Capacity Communication Delay Estimator Bandwidth, Latency Information Which sites to select? 𝑚𝑎𝑥 … view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Bandwidth decreases where as communication delay increases with increasing geographical distances. 0 20 40 60 80 100 120 140 Time (hours) 0.0 0.2 0.4 0.6 Accuracy (%) 10 Workers 20 Workers 30 Workers 40 Workers 50 Workers 60 Workers 70 Workers 80 Workers 90 Workers 100…
Figure 6
Figure 6. Figure 6: Using more sites improves convergence accuracy due to the greater aggregate computational power available, as￾suming the same power and network delay across all sites. Baseline policies. Below, we present the baseline approaches that we compare PowerTrip with. • Random…
Figure 7
Figure 7. Figure 7: Performance metrics - (a) convergence accuracy, (b) time-to-accuracy and (c) speedup factor, improve with more sites, due to greater aggregate computational power available. it provides a holistic measure of end-to-end system performance, capturing the impact of both c…
Figure 8
Figure 8. Figure 8: The static post-hoc policy identifies an optimal num￾ber of sites (in this case, k = 70) that minimizes training time, empirically validating the power-communication trade-off. which prioritizes sites with high power availability and low communication delay. We then ex…
Figure 9
Figure 9. Figure 9: Performance evaluation of PowerTrip’s dynamic policy against the baselines. The existence of this valley in the performance curve empirically shows that an optimal number of sites exists for a given config￾uration. While this offline method is too resource-intensive fo…
Figure 13
Figure 13. Figure 13: Time-to-accuracy comparison across varying model update sizes, demonstrating that PowerTrip consistently out￾performs all baseline policies and highlighting the robustness. PowerTrip remains the most efficient policy, outperforming all baselines. PowerTrip’s dynamic p…
Figure 12
Figure 12. Figure 12: Performance evaluation of PowerTrip’s dynamic policy against the baselines using the Google’s data center production power utilization traces. eventually lead to faster convergence, the immediate, per-round improvement in learning is small at the beginning. This expla…
Figure 14
Figure 14. Figure 14: Time-to-accuracy comparison across different datasets, demonstrating that PowerTrip consistently outper￾forms all baseline policies and highlighting the robustness. confirms that PowerTrip’s adaptive selection strategy is effective for a wide range of training workloa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 50 canonical work pages

  1. [1]

    Gpt-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 Technical Report. arXiv, 2023

    work page 2023

  2. [2]

    Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification

    Saurabh Agarwal, Hongyi Wang, Kangwook Lee, Shivaram Venkataraman, and Dimitris Papailiopoulos. Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification. In Proceedings of Machine Learning and Systems, 2021

    work page 2021

  3. [3]

    Sparse Communication for Distributed Gradient Descent

    Alham Fikri Aji and Kenneth Heafield. Sparse Communication for Distributed Gradient Descent. arXiv, 2017

    work page 2017

  4. [4]

    QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

    Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. NEURIPS, 2017

    work page 2017

  5. [5]

    The AI Disruption: Challenges and Guidance for Data Center Design

    Victor Avelar, Patrick Donovan, Paul Lin, Wendy Torell, and Maria A Torres Arango. The AI Disruption: Challenges and Guidance for Data Center Design. Artificial Intelligence in Medicine, 2023

    work page 2023

  6. [6]

    Keyword Transformer: A Self-Attention Model for Keyword Spotting

    Axel Berg, Mark O’Connor, and Miguel Tairum Cruz. Keyword Transformer: A Self-Attention Model for Keyword Spotting. arXiv, 2021

    work page 2021

  7. [7]

    Flower: A Friendly Federated Learning Research Framework

    Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez- Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane. Flower: A Friendly Federated Learning Research Framework. arXiv, 2020

    work page 2020

  8. [8]

    Towards Federated Learning at Scale: System Design

    Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards Federated Learning at Scale: System Design. In MLSYS, 2019

    work page 2019

Show all 50 references
  1. [9]

    Leaf: A Benchmark for Federated Settings

    Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečn`y, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A Benchmark for Federated Settings. arXiv, 2018

    2018

  2. [10]

    FLAME: Federated Learning across Multi-device Environments

    Hyunsung Cho, Akhil Mathur, and Fahim Kawsar. FLAME: Federated Learning across Multi-device Environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022

    2022

  3. [11]

    EMNIST: Extending MNIST to Handwritten Letters

    Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: Extending MNIST to Handwritten Letters. In Neural Networks, 2017

    2017

  4. [12]

    A Snapshot of the Frontiers of Client Selection in Federated Learning

    Gergely Dániel Németh, Miguel Ángel Lozano, Novi Quadrianto, and Nuria Oliver. A Snapshot of the Frontiers of Client Selection in Federated Learning. arXiv, 2022

    2022

  5. [13]

    Large Scale Distributed Deep Networks

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large Scale Distributed Deep Networks. NEURIPS, 2012

    2012

  6. [14]

    Exponential Laws of Computing Growth

    Peter J Denning and Ted G Lewis. Exponential Laws of Computing Growth. Commun. ACM, 2016

    2016

  7. [15]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NEURIPS, 2023

    2023

  8. [16]

    Beyond A Single AI Cluster: A Survey of Decentralized LLM Training

    Haotian Dong, Jingyan Jiang, Rongwei Lu, Jiajun Luo, Jiajun Song, Bowen Li, Ying Shen, and Zhi Wang. Beyond A Single AI Cluster: A Survey of Decentralized LLM Training. arXiv, 2025

    2025

  9. [17]

    DiLoCo: Distributed Low-Communication Training of Language Models

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. DiLoCo: Distributed Low-Communication Training of Language Models. arXiv, 2023

    2023

  10. [18]

    Client Selection in Federated Learning: Principles, Challenges, and Opportunities

    Lei Fu, Huanle Zhang, Ge Gao, Mi Zhang, and Xin Liu. Client Selection in Federated Learning: Principles, Challenges, and Opportunities. Internet of Things Journal, 2023

    2023

  11. [19]

    TensorFlow Federated: Machine Learning on Decentralized Data

    Google. TensorFlow Federated: Machine Learning on Decentralized Data. https: //www.tensorflow.org/federated, 2019

    2019

  12. [20]

    Efficiency: How We Do It

    Google. Efficiency: How We Do It. https://www.google.com/about/datacenters/ efficiency/, 2024

    2024

  13. [21]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022

    2022

  14. [22]

    ThunderServe: High-Performance and Cost-Efficient LLM Serving in Cloud Environments

    Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, and Eiko Yoneki. ThunderServe: High-Performance and Cost-Efficient LLM Serving in Cloud Environments. arXiv, 2025

    2025

  15. [23]

    MegaScale: Scaling Large Language Model Training to More than 10,000 GPUs

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. MegaScale: Scaling Large Language Model Training to More than 10,000 GPUs. NSDI, 2024

    2024

  16. [24]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv, 2020

    2020

  17. [25]

    Federated Learning: Strategies for Improving Communication Efficiency

    Jakub Konečn `y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated Learning: Strategies for Improving Communication Efficiency. arXiv, 2016

    2016

  18. [26]

    Oort: Efficient Federated Learning via Guided Participant Selection

    Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient Federated Learning via Guided Participant Selection. In USENIX, 2021

    2021

  19. [27]

    Breaking barriers to data center growth

    Vivian Lee, Pattabi Seshadri, Clark O’Niell, Archit Choudhary, Braden Holstege, and Stefan A Deutscher. Breaking barriers to data center growth. Boston Con- sulting Group, 2025

    2025

  20. [28]

    Federated Optimization in Heterogeneous Networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization in Heterogeneous Networks. Proceedings of Machine learning and systems , 2020

    2020

  21. [29]

    Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

    Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. arXiv, 2017

    2017

  22. [30]

    Ecolearn: Optimizing the Carbon Footprint of Federated Learning

    Talha Mehboob, Noman Bashir, Jesus Omana Iglesias, Michael Zink, and David Irwin. Ecolearn: Optimizing the Carbon Footprint of Federated Learning. arXiv, 2023

    2023

  23. [31]

    Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge

    Takayuki Nishio and Ryo Yonetani. Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge. In ICC, 2019

    2019

  24. [32]

    Rethinking Load Growth: Assessing the Potential for Integration of Large Flexible Loads in US Power Systems

    Tyler Norris, Timothy Profeta, Dalia Patino-Echeverri, and Adam Cowie-Haskell. Rethinking Load Growth: Assessing the Potential for Integration of Large Flexible Loads in US Power Systems. 2025

    2025

  25. [33]

    Carbon Emissions and Large Neural Network Training

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon Emissions and Large Neural Network Training. arXiv, 2021

    2021

  26. [34]

    Expanding Data Center Capacity to Meet Growing Demand.McKinsey and Company, 2024

    AI Power. Expanding Data Center Capacity to Meet Growing Demand.McKinsey and Company, 2024

    2024

  27. [35]

    A Generic Framework for Privacy Preserving Deep Learning

    Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner, Jason Mancuso, Daniel Rueckert, and Jonathan Passerat-Palmbach. A Generic Framework for Privacy Preserving Deep Learning. arXiv, 2018

    2018

  28. [36]

    AI, Data Centers and the Coming US Power De- mand Surge

    Goldman Sachs. AI, Data Centers and the Coming US Power De- mand Surge. https://www.goldmansachs.com/pdfs/insights/pages/generational- growth-ai-data-centers-and-the-coming-us-power-surge/report.pdf, 2024

    2024

  29. [37]

    Data Center Power Oversubscription with a Medium Voltage Power Plane and Priority-Aware Capping

    Varun Sakalkar, Vasileios Kontorinis, David Landhuis, Shaohong Li, Darren De Ronde, Thomas Blooming, Anand Ramesh, James Kennedy, Christopher Malone, Jimmy Clidaras, et al. Data Center Power Oversubscription with a Medium Voltage Power Plane and Priority-Aware Capping. In ASPLOS, 2020

    2020

  30. [38]

    Photon: Federated LLM Pre-Training

    Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, et al. Photon: Federated LLM Pre-Training. arXiv, 2024

    2024

  31. [39]

    Green AI

    Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green AI. Com- munications, 2020

    2020

  32. [40]

    A Quantitative Survey of Communication Optimizations in Distributed Deep Learning

    Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Chengjian Liu, Wei Wang, and Bo Li. A Quantitative Survey of Communication Optimizations in Distributed Deep Learning. IEEE Network, 2020

    2020

  33. [41]

    GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions

    Tianyao Shi, Yanran Wu, Sihang Liu, and Yi Ding. GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions. arXiv, 2024

    2024

  34. [42]

    Democratizing AI: Open-Source Scalable LLM Training on GPU-Based Supercomputers

    Siddharth Singh, Prajwal Singhania, Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain, Abhimanyu Hans, Manli Shu, Aditya Tomar, et al. Democratizing AI: Open-Source Scalable LLM Training on GPU-Based Supercomputers. In SC24, 2024

    2024

  35. [43]

    ML Training with Cloud GPU Shortages: Is Cross-Region the Answer? In MLSys, 2024

    Foteini Strati, Paul Elvinger, Tolga Kerimoglu, and Ana Klimovic. ML Training with Cloud GPU Shortages: Is Cross-Region the Answer? In MLSys, 2024

    2024

  36. [44]

    Fusionllm: A Decentralized LLM Training System on Geo-Distributed GPUs with Adaptive Compression

    Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, et al. Fusionllm: A Decentralized LLM Training System on Geo-Distributed GPUs with Adaptive Compression. arXiv, 2024

    2024

  37. [45]

    LlaMa 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. LlaMa 2: Open Foundation and Fine-Tuned Chat Models. arXiv, 2023

    2023

  38. [46]

    Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

    Pete Warden. Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. arXiv, 2018

    2018

  39. [47]

    Sus- tainable AI: Environmental Implications, Challenges and Opportunities

    Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sus- tainable AI: Environmental Implications, Challenges and Opportunities. MLSYS, 2022

    2022

  40. [48]

    SkyPilot: An Intercloud Broker for Sky Computing

    Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, et al. SkyPilot: An Intercloud Broker for Sky Computing. In NSDI, 2023

    2023

  41. [49]

    Openfedllm: Training Large Language Models on Decentralized Private Data via Federated Learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training Large Language Models on Decentralized Private Data via Federated Learning. In SIGKDD, 2024

    2024

  42. [50]

    OPT: Open Pre-Trained Transformer Language Models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open Pre-Trained Transformer Language Models. arXiv, 2022. 13

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.