Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Overlapping computation and communication in distributed GPU training slows compute kernels by 18.9% on average and up to 40% versus an ideal scenario, while still beating sequential execution by about 10.2%.

desk verdict Overlap is not free: a useful but incomplete measurement study whose headline slowdown numbers hinge on an unexplained isolated-compute baseline. read the letter →

arxiv 2507.03114 v1 pith:HZGWGY6N submitted 2025-07-03 cs.DC

Seonho Lee , Jihwan Oh , Junkyum Kim , Seokjin Go , Jongse Park , Divya Mahajan This is my paper · ORCID
classification cs.DC
keywords compute-communicationoverlapdistributedtrainingGPUpowerconsumptionresourcecontentioncappingfullyshardeddataparallelismpipelineperformancecharacterization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed training frameworks treat compute-communication overlap as a free optimization and push it as aggressively as possible. This paper tests that assumption by measuring what actually happens on NVIDIA H100/A100 and AMD MI250/MI210 GPUs when NCCL/RCCL communication runs concurrently with compute kernels during FSDP and pipeline-parallel training of GPT-3 and LLaMA models. The central result is that overlap is not free: compute kernels slow down by an average of 18.9% (up to 40%) compared with an ideal scenario where communication does not perturb compute, because compute and communication contend for shared memory bandwidth and power. Overlap still beats sequential execution, which is on average 10.2% slower, but the gap between real overlap and the ideal makes the hidden cost visible. Power capping amplifies the slowdown, with execution time rising by up to 107% under a strict power cap, so the paper argues that maximizing overlap is not automatically the right policy.

What carries the argument

The load-bearing construct is the slowdown equation and the ideal-time counterfactual built on it. Compute slowdown is defined as $\mathrm{Slowdown} = (\mathrm{ComputeOverlapping} - \mathrm{ComputeSequential}) / \mathrm{ComputeSequential}$, comparing the summed compute kernel times during overlapped execution with a baseline of the same compute running in isolation; from that, the paper defines $E2E_{\mathrm{Ideal}} = E2E_{\mathrm{Overlapping}} - (\mathrm{ComputeOverlapping} - \mathrm{ComputeSequential})$, a hypothetical execution time in which communication hides fully behind compute without perturbing it. The argument runs on kernel-level timings from the PyTorch profiler and torch.cuda.event, plus power traces from NVML (100 ms sampling) and AMD-SMI (20 ms), across FSDP (DeepSpeed/ZeRO) and pipeline parallelism (Megatron-LM). Controlled ablations — power caps from 300W down to 100W, FP32 versus FP16 precision, FP32 versus TF32 tensor-core paths, and a microbenchmark that runs a 1 GB all-reduce concurrently with matrix multiplication — are what let the paper attribute the slowdown to contention for memory bandwidth and the power budget rather than to communication latency alone.

What would settle it

Run one GPT-3 13B FSDP training step on four H100s and measure compute kernel time in three variants: normal overlap; the same computation with NCCL collectives started on a dedicated stream and deliberately delayed; and the same computation with collectives removed so no bytes move. If the compute time in the third variant differs from compute time when communication is merely delayed, the 18.9% slowdown is an artifact of the isolation baseline rather than a genuine contention effect. Separately, inject a known extra communication delay and check whether the computed $E2E_{\mathrm{Ideal}}$ shifts by exactly that delay; if it does not, the ideal-gap numbers are mis-estimated.

Watch

Extended reading notes

Core claim

The paper claims that overlapped execution of communication collectives and compute kernels degrades the compute kernels themselves, and that this degradation is measurable and configuration-dependent. Across GPT-3 and LLaMA models up to 13B parameters on four GPU types, compute-under-overlap runs on average 18.9% slower (maximum 40.0%) than the same compute with no communication happening, which the paper takes as the ideal scenario; meanwhile, running compute and communication strictly sequentially is on average 10.2% slower (maximum 26.6%) than overlap, so overlap still wins end to end. The ideal execution time $E2E_{\mathrm{Ideal}}$ is a constructed quantity, defined as the overlapped end-to-end time minus the measured compute slowdown, so the gap between overlapped and ideal execution is derived rather than directly observed. The paper attributes the slowdown to resource contention: overlap raises peak power by up to 25% (reaching 140% of TDP on H100 for large models), and strict power caps turn that contention into severe slowdowns. It further claims that distribution strategy matters — FSDP's all-gather and reduce-scatter collectives slow compute more than pipeline send/receive patterns — and that FP16 and tensor-core datapaths relieve contention for small models but intensify it for large ones.

Load-bearing premise

Everything hinges on the assumption that the profiling timings correctly separate compute from communication, and that the 'compute with no communication' comparison faithfully represents the same training step — yet the paper does not say how that no-communication baseline is produced in FSDP or pipeline parallelism, where communication is normally mandatory.

Editorial extensions

If this is right

  • Framework designers should treat overlap as a tunable cost rather than a free win: the average 18.9% compute slowdown, reaching near 40% for large sharded models, sets a concrete target for smarter overlap scheduling.
  • Overlap remains the right default: sequential execution is on average 10.2% slower end-to-end, so the practical goal is shrinking the gap between overlapped and ideal execution, not eliminating overlap.
  • Power-capping policies must account for overlap: under a strict cap on A100 systems, overlapped execution time rose by up to 107%, so energy-saving limits that ignore communication-driven power spikes can backfire.
  • Parallelism choice now carries an overlap-cost dimension: FSDP-style collectives (all-gather, reduce-scatter) induce higher compute slowdowns than pipeline send/receive, so contention cost should enter the FSDP-versus-pipeline decision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ideal baseline is never measured directly. A natural test of the whole argument: delay rather than remove communication (e.g., hold NCCL kernels on a separate stream without data movement) and check whether compute kernel times match the no-communication baseline; if they do not, part of the 18.9% figure is a baseline artifact.
  • If contention for memory bandwidth is the true mechanism, the compute slowdown should track bandwidth utilization during overlap. A microbenchmark sweep varying all-reduce size against matrix size would map that contention curve and separate bandwidth from power effects — something the single 1 GB all-reduce microbenchmark only sketches.
  • The single-node scope (NVLink, Infinity Fabric) likely understates the effect: slower cluster interconnects stretch communication over more compute and may push the average slowdown above 18.9% in multi-node training, which the paper does not cover.
  • The results point to a scheduler design the paper leaves implicit: instead of maximizing overlap ratio, dynamically throttle or shift communication to avoid peak-power and bandwidth contention during compute-critical phases — the power-cap data is indirect evidence that such shaping would pay off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript reports an empirical characterization of compute-communication overlap in distributed GPU training on NVIDIA H100/A100 and AMD MI250/MI210. Using FSDP and pipeline-parallel GPT/LLaMA workloads, it measures compute-kernel slowdowns, end-to-end iteration times, and power consumption. The central quantitative claims are that overlapping compute and communication slows compute kernels by 18.9% on average (up to 40%) relative to an ideal no-contention baseline, while remaining 10.2% faster than sequential execution, and that overlap raises peak power and worsens slowdown under power caps. The paper concludes that aggressive overlap is not free and that contention should be accounted for in distributed training systems.

Significance. If the measurement baseline is valid, this is a valuable contribution: it challenges the default assumption that overlap is strictly beneficial, provides cross-vendor evidence (NVIDIA and AMD), and directly measures kernel-level slowdown and power effects with 25-run averaging. The qualitative contention result is corroborated by the power-capping and microbenchmark experiments, which do not depend on the contested ideal baseline. The paper would be strengthened by a reproducible artifact and clearer statistical reporting; as written, the exact percentages rest on an incompletely specified counterfactual.

major comments (3)
  1. [IV-D, Eq. (1)] The paper never states how ComputeSequential is produced. In FSDP and pipeline parallelism, all-gather/reduce-scatter and send/receive are semantically required before subsequent compute, so "compute executed in isolation" cannot be obtained by removing communication without changing tensor sharding or the kernel list. If ComputeSequential is instead derived from the overlapped trace by selecting non-overlapped compute intervals, it is a different counterfactual containing cross-stream dependencies. Because Eq. (1) and Eq. (4) both use ComputeSequential, every headline percentage (18.9%, 40%, 10.2%, 26.6%) inherits this ambiguity. Please describe the exact protocol, show a sample kernel trace for each baseline, and justify why it represents the same computation.
  2. [IV-D, Eqs. (3)-(5)] E2EIdeal is not independently measured; it is defined as E2EOverlapping minus the measured compute slowdown. Consequently, the "gap" between overlapping and ideal execution is partly a construction, and the statement in Section V-A that overlapping execution is 45% slower than ideal on MI250 with GPT-3 13B is a restatement of the slowdown definition rather than an independent observation. Also Eq. (5) defines E2ESequential from E2EIdeal, while Section IV-D says E2ESequential is measured. Clarify which quantities are measured and which are derived, and validate E2EIdeal against a separate no-contention configuration, for example a microbenchmark with and without concurrent communication.
  3. [IV-D and V] Despite averaging over 25 runs, no variance, confidence intervals, or error bars are provided for any of the performance or power figures. With power sampled at 100 ms (NVML) or 20 ms (AMD-SMI), the claims of 25% higher peak power and 107% slowdown under power caps need per-configuration distributions to be interpretable, especially given the heterogeneity across GPUs and models. Please report standard deviations or confidence intervals for the key metrics and clarify the exact sampling setup, including the apparent discrepancy between the 20 ms AMD-SMI interval stated in Section IV-D and the 1 ms granularity mentioned in Section V-B.
minor comments (4)
  1. [IV-D] The sentence about AMD-SMI contains a typo: "AMD-SMI was utilize" should read "AMD-SMI was utilized."
  2. [V] The sentence beginning "As illustrated in Figure 6, provides additional insights" is ungrammatical and should be revised, for example to "Figure 6 provides additional insights into the energy implications."
  3. [IV-D and V] Notation is inconsistent: ComputeOverlapping and ComputeOverlap are both used, as are E2EOverlapping and E2EOverlapping. Please choose one notation per quantity and use it throughout.
  4. [Figures 4-11] The figure captions do not consistently define axis labels and units, especially for normalized power and for slowdown percentages. Adding explicit axis descriptions or table-based data would improve reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

The measured compute-slowdown claims are not fitted, but the paper's 'ideal execution' reference is constructed by subtracting that same measured slowdown (Eq. 4), so the overlapped-vs-ideal gap is an algebraic restatement of Eq. 1 rather than independent evidence.

  1. self definitional [Section IV-D, Equations 3-4; interpreted in Section V-A]
    "They are measured (E2E Overlapping ,ComputeOverlapping ,ComputeSequential) and subsequently computed (E2EIdeal) as: SlowdownCompute = ComputeOverlapping − ComputeSequential (3) E2EIdeal = E2EOverlapping − SlowdownCompute (4). E2EIdeal is a hypothetical time, in which we do not observe the slowdown due to any resource contention, but compute and communication can happen concurrently."

    E2EIdeal is not an independent measurement or counterfactual: it is defined as the overlapped end-to-end time minus the very compute slowdown (ComputeOverlapping − ComputeSequential) whose magnitude is the paper's headline result. Consequently, the reported 'gap' between overlapped and ideal execution, including the Section V-A example of 'overlapping execution exhibited a 45% higher execution time compared to the ideal execution time,' is identically equal to the Eq. 1 slowdown by construction. The ideal comparison therefore cannot validate the claim that resource contention slows compute; it merely restates the input slowdown in end-to-end units.

full rationale

The central 18.9% average and 40% maximum compute-slowdown figures are direct kernel-time comparisons and are not fitted from E2EIdeal; if ComputeSequential was genuinely measured in a communication-free run, the core slowdown claim has independent empirical content. The paper does not specify how that isolation baseline is produced in FSDP or pipeline parallelism, where communication is semantically required, but that is a reproducibility gap rather than demonstrated circularity. What is demonstrably circular is the construction of E2EIdeal in Eq. 4: subtracting the measured slowdown from the overlapped time guarantees that overlapped execution will 'fall short' of the resulting ideal, so that particular comparison is true by definition. There are no load-bearing self-citations or ansatz-smuggling citations in the manuscript; the author-overlapping references are used only as related-work examples. The score reflects one definitional construction that inflates a measured quantity into an 'ideal gap,' while the main characterization claims still rest on measurements that are not, on the evidence quoted, circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central claim rests on kernel profiling and a counterfactual baseline, not on fitted parameters. No free parameters are fit to data. The main constructed quantity is E2EIdeal, which is defined from the measured slowdown and therefore cannot independently corroborate the slowdown claim.

assumptions (4)
  • domain assumption PyTorch profiler and torch.cuda.event correctly separate and time compute and communication kernels.
    Section IV-D: all headline slowdowns rely on this attribution; profiler overhead or misattribution would bias the numbers.
  • domain assumption The isolated ComputeSequential baseline is a valid no-communication counterfactual for the same training step.
    Section IV-D and Eq. 1: the paper states compute is executed in isolation, but does not describe how this mode is produced in FSDP or pipeline-parallel training where communication is normally mandatory.
  • ad hoc to paper E2EIdeal = E2EOverlapping - (ComputeOverlapping - ComputeSequential) gives the true concurrent no-contention execution time.
    Eq. 4: this is a constructed counterfactual, not measured; it assumes communication is fully hidden and the only interaction effect is the measured compute slowdown.
  • domain assumption Power samples from NVML at 100 ms and AMD-SMI at 20 ms capture the peak and average power relevant to overlap.
    Section IV-D: sampling resolution, especially the 100 ms interval on NVIDIA, can miss short power spikes that matter for contention analysis.
invented entities (1)
  • E2EIdeal, the ideal execution time
    purpose: A benchmark for quantifying the cost of overlap by representing compute and communication running concurrently without interference.
    Defined in Eq. 4 as E2EOverlapping minus the measured compute slowdown, so it is a constructed reference point with no independent measurement or external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications." pith.science (2026). https://pith.science/paper/HZGWGY6N

@misc{pith2026250703114,
  author       = {Pith},
  title        = {Pith review of: Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HZGWGY6N}},
  note         = {Machine review of arXiv:2507.03114}
}
read the original abstract

This paper provides an in-depth characterization of GPU-accelerated systems, to understand the interplay between overlapping computation and communication which is commonly employed in distributed training settings. Due to the large size of models, distributing them across multiple devices is required. Overlapping strategies, which enable concurrent computation and communication, are critical for mitigating communication bottlenecks and maximizing GPU utilization. However, the current consensus is that we should always and aggressively overlap compute and communication to mitigate the overhead of distribution. By systematically evaluating state-of-the-art GPUs, this study investigates the impact of hardware features such as numeric precision, specialized cores, and power capping on distributed training workloads. Comprehensive experiments and studies showcase the effects of overlapping strategies on performance and power consumption across varying scenarios. We observe that overlapping computation and communication can result in an average computational slowdown of 18.9%, with a maximum of 40.0% slowdown. This slowdown is in comparison to the scenario when no communication was happening with the compute. We consider this an ideal execution scenario, where the communication in parallel has not impact on the compute time. However, performing computation and communication sequentially is, on average, 10.2% slower than overlapped execution, with a maximum slowdown of 26.6%. We further observe, while specialized datapath and optimized numeric precision mitigate certain slowdowns, overlapping execution can lead to resource contention and also increase power consumption under specific configurations. The analysis also uncovers trade-offs introduced by power and frequency capping, emphasizing the importance of balanced strategies to optimize energy efficiency and training throughput.

Figures

Figures reproduced from arXiv: 2507.03114 by the authors.

Figure 1
Figure 1. Amount of overlapping computation and communica [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. GPU server architecture, illustrating (a) the internal GPU design and (b) the interconnect setup in H100 DGX Box [16]. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of distributed training techniques: (a) Fully Sharded Data Parallelism (FSDP) and (b) Pipeline Parallelism. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Computation slowdowns across GPUs for various models. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: End-to-end training iteration latency across GPUs for various models. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Power consumption across GPUs for various models. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Power trace of MI250 during LLaMA2 13B training. Power is normalized to the TDP, and time is normalized to the [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Power consumption analysis and performance variations with different overlap configurations. A 1GB all-reduce [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Impact of power capping on performance and slowdowns. Evaluated on A100 x 4 systems. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: The effect of numeric precision (FP32 and FP16) on slowdowns and power consumption across different workloads. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Impact of Tensor Core utilization on performance and power consumption for various workloads. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Co-Located AI Training Jobs Synchronize? Load-Dependent Throttling as a Coupling Mechanism for Phase-Locking Behind a Shared Power Cap

    eess.SY 2026-07 conditional novelty 6.0 of 10

    Training jobs sharing a power cap couple like phase oscillators through load-dependent throttling; the coupling is repulsive at fast control delays and attractive beyond half an iteration period, allowing coherent N-s...

Reference graph

Works this paper leans on

61 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020

    work page 2020

  2. [2]

    PipeDream: generalized pipeline parallelism for DNN training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “PipeDream: generalized pipeline parallelism for DNN training,” in ACM Symposium on Operat- ing System Principles (SOSP) , 2019

    work page 2019

  3. [3]

    Zero: Memory optimizations toward training trillion parameter models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” 2020. [Online]. Available: https://arxiv.org/abs/1910.02054

    arXiv 2020

  4. [4]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019

    2019

  5. [5]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

    2020

  6. [6]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.13971

    arXiv 2023

  7. [7]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv .org/abs/2407.21783

    arXiv 2024

  8. [8]

    Pytorch distributed: Experiences on accelerating data parallel training,

    S. Li, Y . Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “Pytorch distributed: Experiences on accelerating data parallel training,” 2020. [Online]. Available: https://arxiv.org/abs/2006.15704

    arXiv 2020

Show all 61 references
  1. [9]

    Deepspeed- inference: Enabling efficient inference of transformer models at unprece- dented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y . He, “Deepspeed- inference: Enabling efficient inference of transformer models at unprece- dented scale,” in SC22: International Conference for High Performan...

    2022

  2. [10]

    Memory-Efficient Pipeline-Parallel DNN Training,

    D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory-Efficient Pipeline-Parallel DNN Training,” in International Conference on Machine Learning (ICML) , 2021

    2021

  3. [11]

    Accelerating recommendation system training by leveraging popular choices,

    M. Adnan, Y . E. Maboud, D. Mahajan, and P. J. Nair, “Accelerating recommendation system training by leveraging popular choices,” Proc. VLDB Endow. , vol. 15, no. 1, p. 127–140, sep 2021. [Online]. Available: https://doi.org/10.14778/3485450.3485462

    2021

  4. [12]

    Heterogeneous acceleration pipeline for recommendation system training,

    ——, “Heterogeneous acceleration pipeline for recommendation system training,” ser. ISCA ’24. IEEE Press, 2024. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00081

    2024

  5. [13]

    Domino: Eliminating communication in llm training via generic tensor slicing and overlapping,

    G. Wang, C. Zhang, Z. Shen, A. Li, and O. Ruwase, “Domino: Eliminating communication in llm training via generic tensor slicing and overlapping,” 2024. [Online]. Available: https://arxiv .org/abs/ 2409.15241

    2024 arXiv

  6. [14]

    NVIDIA Data Center GPUs,

    NVIDIA, “NVIDIA Data Center GPUs,” https://www .nvidia.com/en-us/ data-center/data-center-gpus/, 2024

    2024

  7. [15]

    AMD Instinct Accelerators,

    AMD, “AMD Instinct Accelerators,” https://www .amd.com/en/products/ accelerators/instinct.html, 2024

    2024

  8. [16]

    Nvidia dgx platform,

    Nvidia, “Nvidia dgx platform,” https://www .nvidia.com/en-us/data- center/dgx-platform/, 2024

    2024

  9. [17]

    NVIDIA H100 Tensor Core GPU,

    NVIDIA, “NVIDIA H100 Tensor Core GPU,” 2023. [Online]. Available: https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor- core-gpu-datasheet

    2023

  10. [18]

    NVIDIA A100 TENSOR CORE GPU,

    ——, “NVIDIA A100 TENSOR CORE GPU,” 2021. [Online]. Available: https://www .nvidia.com/content/dam/en-zz/Solutions/Data- Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web .pdf

    2021

  11. [19]

    AMD Instinct™ MI250 Accelerators,

    AMD, “AMD Instinct™ MI250 Accelerators,” 2024. [On- line]. Available: https://www.amd.com/en/products/accelerators/instinct/ mi200/mi250.html

    2024

  12. [20]

    AMD Instinct™ MI210 Accelerators,

    ——, “AMD Instinct™ MI210 Accelerators,” 2024. [On- line]. Available: https://www.amd.com/en/products/accelerators/instinct/ mi200/mi210.html

    2024

  13. [21]

    NVIDIA L4 GPU Accelerator,

    NVIDIA, “NVIDIA L4 GPU Accelerator,” 2023. [Online]. Available: https://www .nvidia.com/content/dam/en-zz/Solutions/Data- Center/l4/PB-11316-001 v01.pdf

    2023

  14. [22]

    NVIDIA T4 TENSOR CORE GPU,

    ——, “NVIDIA T4 TENSOR CORE GPU,” 2019. [Online]. Available: https://www .nvidia.com/content/dam/en-zz/Solutions/Data- Center/tesla-t4/t4-tensor-core-datasheet-951643 .pdf

    2019

  15. [23]

    AMD Instinct™ MI250X Accelerators,

    AMD, “AMD Instinct™ MI250X Accelerators,” 2024. [On- line]. Available: https://www.amd.com/en/products/accelerators/instinct/ mi200/mi250x.html

    2024

  16. [24]

    AMD Instinct™ MI100 Accelerators,

    ——, “AMD Instinct™ MI100 Accelerators,” 2020. [Online]. Available: https://www.amd.com/en/products/accelerators/instinct/mi100.html

    2020

  17. [25]

    NVIDIA H100 Tensor Core GPU Architecture,

    NVIDIA, “NVIDIA H100 Tensor Core GPU Architecture,” https:// resources.nvidia.com/en-us-tensor-core, 2023

    2023

  18. [26]

    Amd cdna 2 architecture,

    AMD, “Amd cdna 2 architecture,” https://www .amd.com/content/dam/ amd/en/documents/instinct-business-docs/white-papers/amd-cdna2- white-paper.pdf, 2022

    2022

  19. [27]

    NVIDIA DGX H100,

    NVIDIA, “NVIDIA DGX H100,” https://resources .nvidia.com/en-us- dgx-systems/ai-enterprise-dgx, 2023

    2023

  20. [28]

    Nvswitch,

    ——, “Nvswitch,” https://www .nvidia.com/en-us/data-center/nvlink/, 2024

    2024

  21. [29]

    Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect,

    A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. R. Tallent, and K. J. Barker, “Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect,” IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 1, p. 94–110, Jan. 2020. [Online]. Available: htt...

    2020

  22. [30]

    AMD Instinct HPC Solutions,

    AMD, “AMD Instinct HPC Solutions,” https://www .amd.com/en/ graphics/servers-instinct-mi-powered-servers, 2024

    2024

  23. [31]

    Understanding data movement in amd multi-gpu systems with infinity fabric,

    G. Schieffer, R. Shi, S. Markidis, A. Herten, J. Faj, and I. Peng, “Understanding data movement in amd multi-gpu systems with infinity fabric,” 2024. [Online]. Available: https://arxiv .org/abs/2410.00801

    2024 arXiv

  24. [32]

    Horovod: fast and easy distributed deep learning in tensorflow,

    A. Sergeev and M. D. Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” 2018. [Online]. Available: https: //arxiv.org/abs/1802.05799 10

    2018 arXiv

  25. [33]

    ImageNet Classification with Deep Convolutional Neural Networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Commun. ACM , vol. 60, no. 6, p. 84–90, May 2017

    2017

  26. [34]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

    2020 arXiv

  27. [35]

    NVIDIA, “Nvlink,” https://developer .nvidia.com/nccl, 2024

    2024

  28. [36]

    Rocm communication collectives library (rccl),

    AMD, “Rocm communication collectives library (rccl),” https:// rocmdocs.amd.com/projects/rccl/en/latest/#, 2024

    2024

  29. [37]

    NVIDIA cuDNN,

    NVIDIA, “NVIDIA cuDNN,” https://developer .nvidia.com/cudnn, 2024

    2024

  30. [38]

    CUTLASS: Fast Linear Algebra in CUDA C++,

    ——, “CUTLASS: Fast Linear Algebra in CUDA C++,” https:// developer.nvidia.com/blog/cutlass-linear-algebra-cuda/, 2017

    2017

  31. [39]

    Generating long sequences with sparse transformers,

    R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” 2019. [Online]. Available: https://arxiv.org/abs/1904.10509

    2019 arXiv

  32. [40]

    Memory-efficient pipeline-parallel DNN training,

    D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory-efficient pipeline-parallel DNN training,” in International Conference on Machine Learning . PMLR, 2021, pp. 7937–7947

    2021

  33. [41]

    GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and Z. Chen, “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” in International Conference on Neural Information Processing Systems (NeuRIPS), 2019

    2019

  34. [42]

    From High-Level Deep Neural Models to FPGAs,

    H. Sharma, J. Park, D. Mahajan, E. Amaro, J. K. Kim, C. Shao, A. Mishra, and H. Esmaeilzadeh, “From High-Level Deep Neural Models to FPGAs,” in International Symposium on Microarchitecture (MICRO), 2016

    2016

  35. [43]

    Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices,

    Y .-H. Chen, T.-J. Yang, J. Emer, and V . Sze, “Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 2, 2019

    2019

  36. [44]

    Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,

    N. P. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, C. Young, X. Zhou, Z. Zhou, and D. Patterson, “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” 2023

    2023

  37. [45]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel,

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y . Hao, A. Mathews, and S. Li, “Pytorch fsdp: Experiences on scaling fully sharded data parallel,”

  38. [46]

    Lancet: Accelerating mixture-of-experts training via whole graph computation- communication overlapping,

    C. Jiang, Y . Tian, Z. Jia, S. Zheng, C. Wu, and Y . Wang, “Lancet: Accelerating mixture-of-experts training via whole graph computation- communication overlapping,” in Machine Learning and Systems (ML- Sys), P. Gibbons, G. Pekhimenko, and C. D. Sa, Eds., 2024

    2024

  39. [47]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y . Xiong, “Tutel: Adaptive mixture-of-experts at scale,” in Machine Learning and Systems (MLSys), 2023

    2023

  40. [48]

    Accelerating distributed MoE training and inference with lina,

    J. Li, Y . Jiang, Y . Zhu, C. Wang, and H. Xu, “Accelerating distributed MoE training and inference with lina,” in Annual Technical Conference (USENIX ATC), 2023

    2023

  41. [49]

    MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Inter- connects,

    H. Kwon, A. Samajdar, and T. Krishna, “MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Inter- connects,” SIGPLAN Not., vol. 53, no. 2, p. 461–475, Mar. 2018

    2018

  42. [50]

    T ABLA : A unified template-based framework for accelerating statistical machine learning,

    D. Mahajan, J. Park, E. Amaro, H. Sharma, A. Yazdanbakhsh, J. Kim, and Hadi Esmaeilzadeh, “T ABLA : A unified template-based framework for accelerating statistical machine learning,” in International Sympo- sium on High-Performance Computer Architecture (HPCA) , 2016

    2016

  43. [51]

    Scale-out acceleration for machine learnng,

    J. Park, H. Sharma, D. Mahajan, J. K. Kim, P. Olds, and Hadi Esmaeilzadeh, “Scale-out acceleration for machine learnng,” in Inter- national Symposium on Microarchitecture (MICRO) , Oct. 2017

    2017

  44. [52]

    In-RDBMS Hardware Acceleration of Advanced Analyt- ics,

    D. Mahajan, J. K. Kim, J. Sacks, A. Ardalan, A. Kumar, and H. Es- maeilzadeh, “In-RDBMS Hardware Acceleration of Advanced Analyt- ics,” 2018

    2018

  45. [53]

    Mixed precision training,

    P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” in International Conference on Machine Learning (ICML), 2018

    2018

  46. [54]

    Automatic Mixed Precision for Deep Learning,

    NVIDIA, “Automatic Mixed Precision for Deep Learning,” https:// developer.nvidia.com/automatic-mixed-precision, 2024

    2024

  47. [55]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I˜nigo Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” 2024. [Online]. Available: https://arxiv .org/abs/2311.18677

    2024 arXiv

  48. [56]

    Polca: Power oversubscription in llm cloud providers,

    P. Patel, E. Choukse, C. Zhang, I. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Polca: Power oversubscription in llm cloud providers,” ArXiv, vol. abs/2308.12908, August 2023. [Online]. Available: https://www.microsoft.com/en-us/research/publication/polca- power-oversu...

    2023 arXiv

  49. [57]

    Prediction-Based power oversubscription in cloud platforms,

    A. G. Kumbhare, R. Azimi, I. Manousakis, A. Bonde, F. Frujeri, N. Mahalingam, P. A. Misra, S. A. Javadi, B. Schroeder, M. Fontoura, and R. Bianchini, “Prediction-Based power oversubscription in cloud platforms,” in Annual Technical Conference (USENIX ATC) , 2021

    2021

  50. [58]

    To- wards energy proportionality for large-scale latency-critical workloads,

    D. Lo, L. Cheng, R. Govindaraju, L. A. Barroso, and C. Kozyrakis, “To- wards energy proportionality for large-scale latency-critical workloads,” in International Symposium on Computer Architecture (ISCA) , 2014

    2014

  51. [59]

    Zeus: Understanding and optimizing GPU energy consumption of DNN training,

    J. You, J.-W. Chung, and M. Chowdhury, “Zeus: Understanding and optimizing GPU energy consumption of DNN training,” in Annual Technical Conference (USENIX ATC), 2023

    2023

  52. [60]

    EnvPipe: Performance- preserving DNN training framework for saving energy,

    S. Choi, I. Koo, J. Ahn, M. Jeon, and Y . Kwon, “EnvPipe: Performance- preserving DNN training framework for saving energy,” in Annual Technical Conference (USENIX ATC), 2023. 11

    2023

  53. [2023]

    Available: https://arxiv .org/abs/2304.11277

    [Online]. Available: https://arxiv .org/abs/2304.11277

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.