Pith. sign in

REVIEW 5 major objections 6 minor 38 references

Energy Consumption in Parallel Neural Network Training

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Energy use in parallel training scales with GPU hours, not GPU count, the paper argues.

desk verdict Empirically useful, but the 'linear scaling' headline is definitional and the energy accounting leaves real gaps; still worth a serious referee. read the letter →

arxiv 2508.07706 v1 pith:WDI7CYCS submitted 2025-08-11 cs.LG cs.AI

Philipp Huber , David Li , Juan Pedro Gutiérrez Hermosillo Muriedas , Deifilia Kieckhefen , Markus Götz , Achim Streit , Charlotte Debus This is my paper
classification cs.LGcs.AI
keywords energyconsumptiondata-paralleltrainingGPUhoursscalingResNet50FourCastNetpowerdrawlargebatcheffects
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that in data-parallel neural network training, energy consumption is set by the number of GPU hours consumed rather than by the number of GPUs or the parallelization strategy as such. Concretely, total energy is approximately a model- and hardware-specific factor times GPU hours: about 0.27 to 0.31 kWh per GPU hour for ResNet50 on A100 GPUs, 0.18 to 0.23 kWh per GPU hour for FourCastNet on A100 GPUs, and 0.26 to 0.28 kWh per GPU hour for FourCastNet on H100 GPUs. The factor is not fixed: it shifts systematically with how many samples and gradient updates pass through each GPU per hour, because those throughputs change GPU power draw. If true, this gives practitioners a simple way to convert planned resource use into an energy budget and shows that time savings from scaling can be partly or fully eaten by higher power draw.

What carries the argument

The load-bearing quantity is the energy-per-GPU-hour factor $\bar E_{\mathrm{GPUh}}$, the mean power draw summed over all devices (GPU, CPU, RAM) averaged over the run. It converts a resource count (GPU hours) into an energy estimate, and its variation is explained by two workload-throughput metrics: samples per GPU hour and forward/backward gradient updates per GPU hour. The paper shows that this factor, not the raw GPU count, is what determines energy, and uses it to compare scaling regimes and hardware.

What would settle it

Repeat one of the paper's fixed-dataset, constant-LBS ResNet runs (for example 32 GPUs) while metering whole-node wall power at sub-second resolution, including network switches, cooling, and H100 RAM, and compare energy against GPU hours across several GPU counts. If the straight-line relation bends or the per-GPU-hour slope moves outside the reported 0.26–0.31 kWh range once the omitted components are included or 1 Hz sampling catches different power spikes, the linear scaling claim is specific to the paper's measurement boundary rather than to training itself.

Watch

Extended reading notes

Core claim

The central discovery is an approximately linear relation between energy and GPU hours across a wide range of data-parallel training configurations. Using ResNet50 on ImageNet-2012 and FourCastNet on ERA5, the authors varied GPU count, local batch size, global batch size, and dataset size, and measured node-level GPU, CPU, and RAM power at 1 Hz. Energy always tracked GPU hours on a straight line, but the slope—energy per GPU hour—depends on the model, the accelerator, and the parallelization parameters. The slope is driven by mean power draw, which is in turn influenced by samples per GPU hour and gradient updates per GPU hour; experiments show GPU power increasing with sample throughput. Th

Load-bearing premise

The result rests on the assumption that the measured energy totals are complete and accurate: power is sampled once per second and summed over GPU, CPU, and RAM, with RAM unmetered on H100 nodes and node-level components such as interconnect, cooling, and power delivery assumed not to change the linear energy-to-GPUh relationship.

Editorial extensions

If this is right

  • Before launching a run, energy can be estimated as a model- and hardware-specific factor times planned GPU hours, with the caveat that the factor shifts with throughput.
  • Adding GPUs does not itself raise energy; it raises energy only through extra GPU hours caused by imperfect speedup, so optimizing speedup is also an energy optimization.
  • Faster hardware does not automatically mean greener training: H100's 25–30% runtime reduction yields less than 5% energy saving because the per-GPU-hour price is higher.
  • Growing dataset size to gain accuracy costs roughly proportional energy, since energy scales about linearly with the number of samples processed.
  • Power profiles matter beyond billing: A100 runs show large power fluctuations that prior work links to shortened hardware lifespan, whereas H100 runs draw steadier power.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear GPU-hour relation generalizes beyond these two models, data centers could meter training energy by multiplying GPU-hour accounting by a learned per-workload factor rather than measuring wall power continuously.
  • The systematic dependence on samples and gradient updates per GPU hour implies that optimizations which raise utilization may raise power draw; the right target is work per joule, so time efficiency and energy efficiency can diverge.
  • The near-tie between H100 and A100 energy suggests a crossover rule: a slower, lower-power accelerator can beat a faster one on energy whenever its power advantage outweighs its runtime penalty—an explicit trade-off users could evaluate per workload.
  • A testable extension is to predict the per-GPU-hour factor from workload statistics such as model, batch geometry, and data-loading rate, then choose parallelization parameters that minimize energy for a target accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper reports scaling experiments for data-parallel training of ResNet50 on ImageNet-2012 and FourCastNet on ERA5, varying GPU count, global batch size (GBS), and local batch size (LBS), and measuring training time, predictive accuracy, and energy consumption with the Perun tool at 1 Hz sampling. Total energy is defined as the sum of GPU, CPU, and RAM energy. The main claimed findings are that energy consumption scales approximately linearly with GPU hours with a model- and hardware-dependent factor (roughly 0.18–0.32 kWh per GPUh), that this factor is systematically influenced by the number of samples and gradient updates per GPU hour, that large-batch effects appear at different GBS for the two models, and that H100 and A100 GPUs exhibit different power profiles and energy per GPUh. The paper includes ResNet runs with five repetitions and one FourCastNet configuration with three repetitions; the remaining FourCastNet scaling runs are single-shot.

Significance. The empirical dataset is valuable: real multi-node HPC measurements, a comparison of A100 and H100, and repeated ResNet runs with standard deviations are strengths. If the quantitative relations are robust, the results could support practical energy budgeting for data-parallel training and inform hardware choices. However, the headline 'linear scaling' is close to a definitional identity, and the claim that throughput metrics 'systematically influence' power is supported only by visual inspection. The FourCastNet scaling results are largely single measurements, and the energy measurement is incomplete for H100 nodes. These limitations currently weaken the generality of the conclusions, though they do not invalidate the measurements themselves.

major comments (5)
  1. [Section 3.5, Energy per GPUh; Conclusion] The linear energy–GPUh relationship is tautological under the paper's definitions. Total energy E is the time integral of summed device powers, and GPUh is N_GPU × time; hence E/GPUh is exactly the time- and device-averaged power per GPU. The empirical content is therefore not that the relationship is linear, but that E/GPUh is roughly constant in certain regimes and varies with configuration. Please state this identity explicitly, report E/GPUh with confidence intervals, and quantify how well a single linear fit describes the full dataset (e.g., R² or residual analysis).
  2. [Section 3.5, Figures 6 and 7] The claim that samples per GPUh and gradient updates per GPUh systematically influence GPU power is not statistically substantiated. Since passes per GPUh = samples per GPUh / LBS, the two metrics are deterministically linked, so separating their effects from the plotted trends alone is not possible. A regression of GPU power on these throughput metrics, with hardware and GPU count as covariates, would be needed. Additionally, the text appears to swap figure numbers: 'ResNet (Figure 7)' and 'FourCastNet (Figure 6)' in the second half of Section 3.5 are inconsistent with the figure captions.
  3. [Sections 3.1 and 3.3, energy measurement] The absolute scaling factors and hardware comparisons rest on the assumption that unmeasured H100 RAM, interconnect, power-supply, and cooling contributions are negligible or constant. No calibration against node-level wall power is provided. Because interconnect power scales with communication volume and PSU/cooling losses scale with load, the omitted component could vary with GBS/LBS/GPU count and bias the reported trends. Please report a sensitivity analysis or, at minimum, present the energies as 'measured GPU+CPU+RAM energy' and quantify coverage on a subset of nodes.
  4. [Section 3.2.2, Figures 2 and 4] There is a systematic top-1 accuracy/error confusion. The definition says 'top-1 error, i.e., the accuracy with which the model predicts the image labels correctly,' and the text says 'top-1 prediction accuracy remains approximately constant at about 28%,' while Figure 2B is labeled 'top-1 error.' For ResNet50 on ImageNet, 28% is plausible as top-1 error, not accuracy. Figure 4B similarly uses 'top-1 error' for the FourCastNet Z500 RMSE metric. This must be corrected throughout because it affects the interpretation of large-batch effects.
  5. [Section 3.3, FourCastNet scaling runs] Except for one 20-epoch configuration with three runs, all FourCastNet scaling experiments are single measurements, yet they support quantitative claims such as the exponential increase in fine-tuning energy and the <5% H100/A100 energy difference in Table 1. With no error bars, these differences and trends are not statistically supported. Please add repeats for at least the key configurations, or clearly label these results as preliminary single-run observations.
minor comments (6)
  1. [Throughout] The text contains many corrupted symbols and equations (e.g., '� � � gradient updates· �GPUs· LBS', '¯�GPUh'), making definitions hard to read. Please provide a clean, properly rendered version.
  2. [Figures 2D, 3D, 7D] For the energy-versus-GPUh plots, reporting fitted slopes and R² values would make the 'approximately linear' claim more precise and comparable across experiments.
  3. [Table 1] The table heading mentions 'energy consumptions per node,' but the table lists only total energy, runtime, and GPUh. Either add per-node energy or remove the phrase.
  4. [Section 3.2.1] The text says 'the root mean square error (RMSE) was computed' over five runs, while the Figure 2 caption says 'standard deviation over five separate runs.' Clarify which statistic is plotted and what RMSE refers to.
  5. [Section 3.4] The statement that H100 GPUs 'are running more efficiently' based on narrower power profiles is speculative; power-profile shape alone does not establish efficiency. Please qualify it or relate it to measured work per energy.
  6. [References] Several references are incomplete, e.g., reference [6] says 'Online available' without a URL or venue. Please complete the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the observed linear energy–GPU-hour scaling is an empirical regularity, not an artifact of definitions or fitted parameters.

full rationale

The paper's central empirical claims—that total energy grows approximately linearly with GPU hours, that the energy-per-GPU-hour factor differs across models and hardware, and that this factor correlates with samples and gradient updates per GPU hour—are derived from independent measurements of wall-clock time, GPU count, and Perun-based energy readings. Section 3.5's statement that the energy per GPU hour equals the sum of the mean device power draws is an accounting identity (average power), but the substantive content—that this quantity remains within a narrow interval and drifts systematically with throughput—is not entailed by that identity; average power could vary arbitrarily. No parameter is fit to a subset of data and then renamed a prediction; no uniqueness theorem is imported from the authors' prior work; and no ansatz is smuggled in via citation. The self-citations present (the Perun measurement tool [37], DASO [33], and a prior Green AI position paper [8]) are methodological or contextual and are not load-bearing for the scaling result. Measurement-completeness concerns such as 1 Hz sampling and unsupported H100 RAM energy are validity threats to the reported numbers, not circularity in the derivation of those numbers. The paper is an empirical measurement study whose scaling observations stand as data-driven findings rather than consequences of definitions.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or model entities are introduced. The paper's central quantitative claims rest on measured power and energy data; the only fitted value is a descriptive linear slope. The main assumptions are about measurement completeness and representativeness of the workloads.

free parameters (1)
  • energy-per-sample slope (ResNet scaled dataset, constant LBS) = ≈ 0.0235 Wh/sample
    Linear fit of measured energy vs. number of samples for the scaled-ImageNet constant-LBS experiment (Section 3.2.3, 'with about 0.0235 Wh per sample'). Used to support the conclusion that larger datasets cost proportionally more energy.
assumptions (3)
  • domain assumption Perun 1 Hz sampling of GPU, CPU, and RAM power accurately captures the energy used by training, and unmeasured components do not materially affect the reported totals.
    Section 3.1 defines total energy as the sum of GPU, CPU, and RAM energy with 1 s sampling; on H100 nodes RAM energy is not measured. All energy and scaling conclusions depend on this measurement completeness.
  • domain assumption The two workloads (ResNet50 on ImageNet, FourCastNet on ERA5) are representative of typical data-parallel neural network training.
    The paper generalizes from these two models to conclusions about energy scaling in data-parallel deep learning (Introduction, Section 3). If these workloads are atypical, the reported scaling factors and trends may not transfer.
  • domain assumption Variation across the five ResNet runs with the same random seed reflects hardware fluctuation, and the RMSE or standard deviation over runs estimates measurement noise.
    Section 3.2.1: 'Each experiment was conducted five times with the same random seed to account for fluctuations in hardware performance.' The error bars assume no algorithmic stochasticity from the seed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy Consumption in Parallel Neural Network Training." pith.science (2026). https://pith.science/paper/WDI7CYCS

@misc{pith2026250807706,
  author       = {Pith},
  title        = {Pith review of: Energy Consumption in Parallel Neural Network Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDI7CYCS}},
  note         = {Machine review of arXiv:2508.07706}
}
read the original abstract

The increasing demand for computational resources of training neural networks leads to a concerning growth in energy consumption. While parallelization has enabled upscaling model and dataset sizes and accelerated training, its impact on energy consumption is often overlooked. To close this research gap, we conducted scaling experiments for data-parallel training of two models, ResNet50 and FourCastNet, and evaluated the impact of parallelization parameters, i.e., GPU count, global batch size, and local batch size, on predictive performance, training time, and energy consumption. We show that energy consumption scales approximately linearly with the consumed resources, i.e., GPU hours; however, the respective scaling factor differs substantially between distinct model trainings and hardware, and is systematically influenced by the number of samples and gradient updates per GPU hour. Our results shed light on the complex interplay of scaling up neural network training and can inform future developments towards more sustainable AI research.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 13 canonical work pages

  1. [1]

    Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,

    T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis, ”ACM Comput. Surv., vol. 52, no. 4, 8 2019, doi:10.1145/3320060

    doi:10.1145/3320060 2019

  2. [2]

    Distributed training of deep neural networks: Theoretical and practical limits of parallel scalability,

    J. Keuper and F.-J. Preundt, “Distributed training of deep neural networks: Theoretical and practical limits of parallel scalability, ” 2016 2nd Work- shop on Machine Learning in HPC Environments (MLHPC) , pp. 19–26, 2016, doi:10.1109/MLHPC.2016.006

    work page doi:10.1109/mlhpc.2016.006 2016

  3. [3]

    Pytorch distributed: Experiences on accelerating data parallel training,

    S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “Pytorch distributed: Experiences on accelerating data parallel training, ” arXiv preprint arXiv:2006.15704 , 2020, doi:10.48550/arXiv.2006.15704

    arXiv doi:10.48550/arxiv.2006.15704 2006

  4. [4]

    Accurate, large minibatch sgd: Training imagenet in 1 hour,

    P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour, ” arXiv preprint arXiv:1706.02677, 2017, doi:10.48550/arXiv.1706.02677

    arXiv doi:10.48550/arxiv.1706.02677 2017

  5. [5]

    Large batch training of convolutional net- works,

    Y. You, I. Gitman, and B. Ginsburg, “Large batch training of convolutional net- works, ”arXiv preprint arXiv:1708.03888, 2017, doi:10.48550/arXiv.1708.03888

    arXiv doi:10.48550/arxiv.1708.03888 2017

  6. [6]

    Yet another accelerated sgd: Resnet-50 training on imagenet in 74.7 seconds,

    M. Yamazaki, A. Kasagi, A. Tabuchi, T. Honda, M. Miwa, N. Fukumoto, T. Tabaru, A. Ike, and K. Nakashima, “Yet another accelerated sgd: Resnet-50 training on imagenet in 74.7 seconds, ” arXiv preprint arXiv:1903.12650 , 2019, doi:10.48550/arXiv.1903.12650

    arXiv doi:10.48550/arxiv.1903.12650 1903

  7. [7]

    On the sdes and scaling rules for adaptive gradient algorithms,

    S. Malladi, K. Lyu, A. Panigrahi, and S. Arora, “On the sdes and scaling rules for adaptive gradient algorithms, ”Advances in Neural Information Processing Systems , vol. 35, pp. 7697–7711, 2022. Online available

    work page 2022

  8. [8]

    Reporting electricity consumption is essential for sustainable ai,

    C. Debus, M. Piraud, A. Streit, F. Theis, and M. Götz, “Reporting electricity consumption is essential for sustainable ai, ”Nat. Mach. Intell., vol. 5, no. 11, pp. 1176–1178, 11 2023, doi:10.1038/s42256-023-00750-1

    work page doi:10.1038/s42256-023-00750-1 2023

Show all 38 references
  1. [9]

    Energy costs of communicating with ai,

    M. Dauner and G. Socher, “Energy costs of communicating with ai, ”Front. Com- mun., vol. 10, 6 2025, doi:10.3389/fcomm.2025.1572947

    2025

  2. [10]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition, ” Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770–778, 2016, doi:10.1109/CVPR.2016.90

    2016 doi

  3. [11]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database, ”2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, 2009, doi:10.1109/CVPR.2009.5206848

    2009

  4. [12]

    Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators,

    J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli et al., “Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators, ” arXiv preprint arXiv:2202.11214, 2...

    2022 doi arXiv

  5. [13]

    Fourcastnet,

    “Fourcastnet, ” NVIDIA Corporation, 2022. Online available

    2022

  6. [14]

    The era5 global reanalysis,

    H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers et al., “The era5 global reanalysis, ”Q. J. R. Meteorol. Soc , vol. 146, no. 730, pp. 1999–2049, 2020, doi:10.1002/qj.3803

    1999 doi

  7. [15]

    Green ai,

    R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai, ” CACM, vol. 63, no. 12, pp. 54–63, 11 2020, doi:10.1145/3381831

    2020 doi

  8. [16]

    Quantifying the carbon emissions of machine learning,

    S. Luccioni, V. Schmidt, A. Lacoste, and T. Dandres, “Quantifying the carbon emissions of machine learning, ” NeurIPS 2019 Workshop on Tackling Climate Change with Machine Learning , 2019. Online available

    2019

  9. [17]

    Towards energy-efficient deep learning: An overview of energy-efficient approaches along the deep learning lifecycle,

    V. Mehlin, S. Schacht, and C. Lanquillon, “Towards energy-efficient deep learning: An overview of energy-efficient approaches along the deep learning lifecycle, ” arXiv preprint arXiv:2303.01980, 2023, doi:10.48550/arXiv.2303.01980

    2023 doi arXiv

  10. [18]

    Energy efficiency of training neural network architectures: an empirical study,

    Y. Xu, S. Martínez-Fernández, M. Martinez, and X. Franch, “Energy efficiency of training neural network architectures: an empirical study, ” arXiv preprint arXiv:2302.00967, 2023, doi:10.48550/arXiv.2302.00967

    2023 doi arXiv

  11. [19]

    Measuring the energy consumption and efficiency of deep neural networks: An empirical analysis and design recommendations,

    C. E. Tripp, J. Perr-Sauer, J. Gafur, A. Nag, A. Purkayastha, S. Zisman, and E. A. Bensen, “Measuring the energy consumption and efficiency of deep neural networks: An empirical analysis and design recommendations, ”arXiv preprint arXiv:2403.08151, 2024, doi:10.48550/arXiv.2403.08151

    2024 doi arXiv

  12. [20]

    Zeus: Understanding and optimizing GPU energy consumption of DNN training,

    J. You, J.-W. Chung, and M. Chowdhury, “Zeus: Understanding and optimizing GPU energy consumption of DNN training, ” 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) , pp. 119–139, 2023. Online available

    2023

  13. [21]

    Precise energy consumption measurements of heterogeneous artificial intelligence workloads,

    R. Caspart, S. Ziegler, A. Weyrauch, H. Obermaier, S. Raffeiner, L. P. Schuh- macher, J. Scholtyssek, D. Trofimova, M. Nolden, I. Reinartzet al., “Precise energy consumption measurements of heterogeneous artificial intelligence workloads, ” International Conference on High Per...

    2022 doi

  14. [22]

    Uncovering energy- efficient practices in deep learning training: Preliminary steps towards green ai,

    T. Yarally, L. Cruz, D. Feitosa, J. Sallou, and A. Van Deursen, “Uncovering energy- efficient practices in deep learning training: Preliminary steps towards green ai, ”2023 IEEE/ACM 2nd International Conference on AI Engineering–Software Engineering for AI (CAIN) , pp. 25–36, ...

    2023

  15. [23]

    Spend more to save more (sm2): An energy-aware implementation of successive halving for sustainable hyperpa- rameter optimization,

    D. Geissler, B. Zhou, S. Suh, and P. Lukowicz, “Spend more to save more (sm2): An energy-aware implementation of successive halving for sustainable hyperpa- rameter optimization, ” 2024

    2024

  16. [24]

    The power of training: How different neural network setups influence the energy demand,

    D. Geißler, B. Zhou, M. Liu, S. Suh, and P. Lukowicz, “The power of training: How different neural network setups influence the energy demand, ”International Conference on Architecture of Computing Systems, pp. 33–47, 2024, doi:10.1007/978- 3-031-66146-4_3

    2024 doi

  17. [25]

    Benchmarking resource usage for efficient distributed deep learning,

    N. C. Frey, B. Li, J. McDonald, D. Zhao, M. Jones, D. Bestor, D. Tiwari, V. Gade- pally, and S. Samsi, “Benchmarking resource usage for efficient distributed deep learning, ”2022 IEEE High Performance Extreme Computing Conference (HPEC) , pp. 1–8, 2022, doi:10.1109/HPEC55821.2...

    2022

  18. [26]

    Performance and energy aware training of a deep neural network in a multi-gpu environment with power capping,

    G. Koszczał, J. Dobrosolski, M. Matuszek, and P. Czarnul, “Performance and energy aware training of a deep neural network in a multi-gpu environment with power capping, ”European Conference on Parallel Processing , pp. 5–16, 2023, doi:10.1007/978-3-031-48803-0_1

    2023 doi

  19. [27]

    Communication-efficient large-scale distributed deep learning: A comprehensive survey,

    F. Liang, Z. Zhang, H. Lu, V. Leung, Y. Guo, and X. Hu, “Communication-efficient large-scale distributed deep learning: A comprehensive survey, ”arXiv preprint arXiv:2404.06114, 2024, doi:10.48550/arXiv.2404.06114. ����� �� ���

    2024 doi arXiv

  20. [28]

    On efficient training of large-scale deep learning models,

    L. Shen, Y. Sun, Z. Yu, L. Ding, X. Tian, and D. Tao, “On efficient training of large-scale deep learning models, ”ACM Comput. Surv., vol. 57, no. 3, 11 2024, doi:10.1145/3700439

    2024 doi

  21. [29]

    PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization,

    T. Vogels, S. P. Karimireddy, and M. Jaggi, “PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization, ” Advances in Neural Information Processing Systems, vol. 32, 2019. Online available

    2019

  22. [30]

    Learned Gradient Compression for Distributed Deep Learning,

    L. Abrahamyan, Y. Chen, G. Bekoulis, and N. Deligiannis, “Learned Gradient Compression for Distributed Deep Learning, ”IEEE Trans. Neural Netw. Learn , vol. 33, no. 12, pp. 7330–7344, 12 2022, doi:10.1109/TNNLS.2021.3084806

    2022

  23. [31]

    HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent,

    F. Niu, B. Recht, C. Re, and S. J. Wright, “HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent, ”arXiv preprint arXiv:1106.5730, 11 2011, doi:10.48550/arXiv.1106.5730

    2011 doi arXiv

  24. [32]

    Don’t Use Large Mini-Batches, Use Local SGD,

    T. Lin, S. U. Stich, K. K. Patel, and M. Jaggi, “Don’t Use Large Mini-Batches, Use Local SGD, ”arXiv preprint arXiv:1808.07217, 2 2020, doi:10.48550/arXiv.1808.07217

    2020 doi arXiv

  25. [33]

    Accelerating neural network training with distributed asynchronous and selective optimization (daso),

    D. Coquelin, C. Debus, M. Götz, F. von der Lehr, J. Kahn, M. Siggel, and A. Streit, “Accelerating neural network training with distributed asynchronous and selective optimization (daso), ”J. Big Data, vol. 9, no. 1, p. 14, 2 2022, doi:10.1186/s40537- 021-00556-1

    2022 doi

  26. [34]

    Efficient data-parallel distributed dnn train- ing for big dataset under heterogeneous gpu cluster,

    S. Ahn, S. Lee, H. Choi, and J. Lee, “Efficient data-parallel distributed dnn train- ing for big dataset under heterogeneous gpu cluster, ” 2024 IEEE International Conference on Big Data (BigData) , pp. 179–188, 2024, doi:10.1145/567752.567774

    2024

  27. [35]

    An empirical study of training self-supervised vision transformers,

    X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers, ”Proceedings of the IEEE/CVF international conference on computer vision, pp. 9640–9649, 2021, doi:10.1109/ICCV48922.2021.00950

    2021

  28. [36]

    Fourcastnet: Accelerating global high- resolution weather forecasting using adaptive fourier neural operators,

    T. Kurth, S. Subramanian, P. Harrington, J. Pathak, M. Mardani, D. Hall, A. Miele, K. Kashinath, and A. Anandkumar, “Fourcastnet: Accelerating global high- resolution weather forecasting using adaptive fourier neural operators, ” Pro- ceedings of the platform for advanced scie...

    2023

  29. [37]

    Perun: Benchmarking energy consumption of high-performance computing applications,

    J. P. Gutiérrez Hermosillo Muriedas, K. Flügel, C. Debus, H. Obermaier, A. Streit, and M. Götz, “Perun: Benchmarking energy consumption of high-performance computing applications, ”European Conference on Parallel Processing , pp. 17–31, 2023, doi:10.1007/978-3-031-39698-4_2

    2023 doi

  30. [38]

    Gpu lifetimes on titan supercomputer: Survival analysis and reliabil- ity,

    G. Ostrouchov, D. Maxwell, R. A. Ashraf, C. Engelmann, M. Shankar, and J. H. Rogers, “Gpu lifetimes on titan supercomputer: Survival analysis and reliabil- ity, ”SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–14, 2020, do...

    2020 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.