Pith. sign in

REVIEW 2 major objections 7 minor 61 references

Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution

T0 review · 2 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DHelix co-schedules one micro-batch's forward pass with another's backward pass, hiding communication cost behind computation and reporting 12-40% speedups on distributed LLM training.

desk verdict A serious systems paper with a genuinely new scheduling idea, broad evaluation, and one unvalidated additivity assumption in the DP planner that should be checked before the numbers are trusted. read the letter →

arxiv 2411.15871 v1 pith:FP53QKYR submitted 2024-11-24 cs.DC

Haiquan Wang , Chaoyi Ruan , Jia He , Jiaqi Ruan , Chengjie Tang , Xiaosong Ma , Cheng Li This is my paper · ORCID
classification cs.DC
keywords distributedLLMtrainingcommunication-computationoverlapmicro-batchco-executionstrandinterleavingmodelfoldingW-shapedpipelinedynamicprogrammingschedulingGPUthroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed training of large language models wastes a large fraction of GPU time on inter-GPU communication, and end-to-end model FLOPS utilization often stays below 50%. DHelix's central claim is that this cost can be hidden by co-executing two micro-batches on each GPU: the forward pass of one micro-batch runs together with the backward pass of another, so the communication operators of one strand overlap with the computation operators of the other, which have no data dependence between strands. A dynamic-programming search, fed by offline pairwise operator-overlap measurements, decides which operator segments to pair, and a model-folding technique reshapes pipeline parallelism into a W-shaped schedule so the two strands share a single copy of model parameters. The paper reports 12-40% throughput gains over the Megatron-LM baseline on a 64-GPU A40 cluster, 2-29% on A800, and up to 27% for the Phi Mixture-of-Experts model, with under 3% extra memory and no change to training semantics. If the claim holds, communication-bound LLM training becomes significantly cheaper without requiring users to change their parallelism settings.

What carries the argument

Strand Interleaving (SI) is the central mechanism: the continuous stream of micro-batches is viewed as two strands, $\alpha$ and $\beta$, and DHelix executes the forward pass of one strand together with the backward pass of the other on the same GPU. The search engine is a dynamic program over operator segments; for forward and backward sequences $S_f$ and $S_b$ partitioned into $N_f$ and $N_b$ segments, the optimal makespan obeys $T_{opt}(i,j)=\min\{T_{opt}(i-1,j)+P(i,\varnothing),\, T_{opt}(i,j-1)+P(\varnothing,j),\, T_{opt}(i-1,j-1)+P(i,j)\}$, where $P(i,j)$ is the overlapped execution time of the $i$-th forward and $j$-th backward segment taken from offline pairwise profiling. The third piece is model folding: a 32-layer model laid out linearly across pipeline stages is folded into a U-shape so each GPU holds two non-adjacent layer segments, which turns the classical 1F1B "V" schedule into a "W" schedule in which both strands travel in the same direction and share one parameter copy.

What would settle it

Take a small transformer layer on one GPU, exhaustively enumerate every valid partition of the forward and backward operator sequences into segments, and measure the actual makespan of each pairing; if the DP-chosen plan from the recurrence is not within measurement noise of the fastest exhaustive plan under realistic memory pressure, the pairwise-cost additivity assumption fails and the optimality claim collapses.

Watch

Extended reading notes

Core claim

On its own terms, DHelix's discovery is that the sequential forward-backward cycle of micro-batch training is not a single stream but can be re-read as two interleavable strands, one moving forward and one moving backward, whose operator-level co-execution fills the idle gaps that intra-batch overlap cannot reach. The paper argues that while computation and communication operators inside one micro-batch are blocked by data dependence, operators from opposite strands have no such dependence, so a forward pass and a backward pass can be cut into segments and paired across strands with inserted barriers. Profiling each ordered pair of compute and communication operators yields an overlap cost, and a dynamic program chooses the pairing that minimizes total makespan. Model folding turns the linear layer assignment of pipeline parallelism into a U-shape, so the two strands traverse the pipeline in the same direction and share one parameter copy rather than two. The paper's conclusion is that this construction hides most of the visible communication cost, producing its reported throughput gains and keeping the largest trainable model within 2.5% of the single-strand memory limit.

Load-bearing premise

The search treats the measured time of each overlapped operator pair as a fixed number that can be added up independently for every segment, regardless of memory pressure from the other strand or interference from earlier kernels on the same streams; if those pairwise costs are not additive, the chosen plan may not be the fastest one.

Editorial extensions

If this is right

  • On clusters whose inter-GPU links are slow relative to compute, DHelix's reported gain is 12-40% for dense Llama/GPT models and up to 27% for the Phi MoE model, with the largest gains appearing where communication dominates execution time.
  • DHelix composes with existing DP, TP, SP, CP, and EP parallelism as a layer underneath them, and with pipeline parallelism via the W-shaped folding, so a training job can enable it without changing user-level parallelism parameters.
  • Cross-node tensor parallelism becomes practical on fast clusters: scaling TP from 8 to 32 loses much less per-GPU throughput with DHelix than with the baseline, because cross-node AllGather and ReduceScatter traffic is hidden behind computation.
  • Because the two strands share one copy of model states and their activation footprints interleave, DHelix can run two micro-batches per GPU with under 3% extra memory, and the maximum supported model size stays within 2.5% of the single-strand memory limit.
  • Training semantics and convergence are unchanged, since DHelix only reorders and overlaps operators rather than altering the optimization loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The DP optimality claim rests on pairwise overlap costs $P(i,j)$ being additive across segments; a direct test would be to compare the DP-chosen plan's predicted makespan against an exhaustive search on a small layer, and against actual execution with competing NCCL traffic.
  • The same strand-interleaving machinery could apply to other workloads whose operators have complementary resource usage, such as inference serving with batched prefill and decode, or hybrid data-movement/compute kernels; the paper sketches this generality but does not evaluate it.
  • The memory-sharing benefit relies on timing the forward's activation allocation against the backward's release; with longer sequences or micro-batch sizes that break the complementary triangle, the claimed under-3% overhead could degrade, which is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper presents DHelix, a distributed LLM training micro-structure that co-executes two micro-batches per GPU by interleaving the forward pass of one strand with the backward pass of another. The central mechanism, Strand Interleaving (SI), partitions each strand's operator sequence for a transformer layer into segments and uses a dynamic-programming search over segment pairings, fed by offline pairwise operator-overlap measurements (Section 3.2), to produce a co-scheduling plan that is executed on three CUDA streams (computation, local-node communication, cross-node communication) synchronized by cross-strand barriers. To make SI compatible with pipeline parallelism, the paper introduces model folding: the linear layer layout is folded into a U-shape so that the two strands traverse the GPUs in the same direction, yielding a W-shaped pipeline schedule that shares a single parameter copy. The paper reports 27-40% (Llama) and 26-40% (GPT) throughput gains over Megatron-LM on a 64-GPU A40 cluster, 7-24% on A800, up to 17% on H100 with cross-node TP, and 15-27% on Phi MoE models, with a measured maximum supported model size of 39B versus 40B (2.5% overhead). The design preserves per-strand data dependencies, so training semantics are claimed to be unchanged. The evaluation spans three clusters, three model families, and the DP/TP/PP/CP/EP parallelism dimensions.

Significance. If the results are reproducible, this is a substantial systems contribution: the W-shaped folding technique cleanly removes the model-replication barrier that limited prior inter-batch schemes under PP, the memory claim is measured rather than asserted (39B vs. 40B, Section 6.5), and the evaluation is unusually broad for the stated hardware access window. The reported speedups are end-to-end measurements, and the DP inputs come from a separate offline profiling step rather than from fitting to the evaluated workloads, so I see no circularity between the profiling and the headline numbers; the public release of the pairwise overlap tables (Ref. 12) is a further strength. My reservations are about robustness rather than method: the DP optimality claim rests on an unvalidated additive-cost model (Section 4.2, Eq. (2)), and the evaluation has no Domino comparison, no repeated runs for the small-margin configurations, and no validation that the author-reimplemented baselines reproduce the original systems' behavior.

major comments (2)
  1. [§4.2, Eq. (2)] The DP recurrence in Eq. (2) treats P(i,j), the overlapped execution time of one forward segment and one backward segment, as a fixed and additive cost, so that the optimal makespan is a sum of selected pairwise costs. The paper neither specifies how P(i,j) is derived for a multi-operator segment from the per-operator OEF table (Table 2, Figure 4) nor validates that the cost of a segment pair is unchanged when it is embedded in a longer schedule along with the other strand and previously launched kernels. This concern is grounded in the paper's own measurements: Table 7 reports a 20-30% slowdown of kernels when computation and communication run concurrently on the three streams, and a 10-20% launch-interval overhead, both of which are schedule-context effects that isolated pairwise profiling cannot capture. Under the runtime of Section 5, kernels of the next segment are already queued on other streams while the current pair executes, and default-stream memory allocation interleaves with all three streams, so additivity is likely violated in practice. The claim that DP finds the optimal operator pairing plan is therefore unsupported, and the generality of the 12-40% gains in Figures 11-14 is in question. Please (a) state explicitly how segment-level P(i,j) is computed from the operator profiles, (b) validate the model by comparing DP-predicted makespan with measured makespan for the plans shown in Figures 10 and 16, and (c) show that the end-to-end gains are insensitive to the DP weights, for example by testing random or perturbed weights or by comparing a small set of alternative plans.
  2. [§6.1–§6.3, Figures 13–14] The evaluation omits a comparison with Domino [51], the closest recent work on communication elimination and overlap in LLM training, which is cited only in §6.3.3; for a paper claiming to significantly outperform state-of-the-art methods, this is a gap that should be filled or explicitly justified. In addition, the Intra-batch and Wavelet+ baselines are author reimplementations (§6.1), and no validation is reported that they faithfully reproduce the original systems' behavior. Finally, all throughput numbers appear to be single runs with no error bars, while several claimed margins are small: 1.03× at TP8 in Figure 14(a), 1.05× at TP8 in Figure 14(b), and 1.09× in Figure 13(a). These margins are within typical run-to-run variance for distributed training, so the claims of improvement in the low-communication regime are not statistically established; the footnote in §6.3.3, which admits differing layer counts on H100, further limits the cross-node TP comparison. I ask for at least three repeated runs for these configurations with means and variance reported, and for either a Domino comparison or a clear statement of why it is out of scope.
minor comments (7)
  1. [§4.2/Figure 9] The operator sequencing and partitioning are described at the granularity of one transformer layer (forward/backward DAG), but the paper does not state how the per-layer pairing plans compose across the W-shaped pipeline, where each GPU hosts two layer groups and is visited twice per pass; please clarify whether barriers are inserted at layer boundaries as well as segment boundaries and how the plan is instantiated for both layer groups on a GPU.
  2. [§6.1] The testbed paragraph mentions an A100 cluster with 8 A100 GPUs, but no A100 results appear anywhere in the evaluation; this appears to be leftover text from an earlier version.
  3. [§4.3 vs. §6.1] The paper refers to the NVIDIA A40 as 40GB in §4.3 (Figure 10 caption) but as 48GB in §6.1; the memory capacity should be reported consistently.
  4. [Figures 11–14] The captions print speedup ratios without stating the reference bar; for example, Figure 13(a) lists four ratios (1.02×, 1.07×, 1.09×, 1.24×) for what the text describes as two configurations and two baselines, which is ambiguous. Please state in each caption that the ratios are relative to Megatron-LM, or specify the reference bar explicitly.
  5. [§4.2] No complexity or runtime figures are given for the enumeration of topological orderings and the DP search; since the evaluation covers many parallelism configurations, please state the search cost and clarify whether the 10-30 minute profiling estimate includes the search itself.
  6. [§6.1] The assertion that DHelix does not affect convergence or accuracy because it preserves training semantics is sound by construction, but a single loss or training-curve check for one configuration would make the claim concrete.
  7. [§4.1, Figure 8] The statement that the bubble ratio remains at p/(m−1) needs a derivation: with the U-shaped folding, each micro-batch traverses the pipeline twice per pass (the paper acknowledges the doubled Send/Recv volume), so the effective stage count is 2p, and the standard 1F1B bubble expression is (p−1)/(m+p−1) rather than p/(m−1). Either derive the bubble fraction for the W-shaped schedule or measure it, since the paper's narrative attributes DHelix's gains entirely to SI block shortening rather than to bubble reduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DHelix's speedups are end-to-end measurements, and the DP search uses profiling as input rather than fitting to the target.

full rationale

The paper's derivation chain is self-contained. The only optimization input is operator-level pairwise overlap profiling (OEF in Eq. 1 and P(i,j) in Eq. 2), which is used to select a strand-interleaving plan via dynamic programming. This is a standard use of profiling to construct a scheduler, not a fit to the reported end-to-end throughputs. The claimed 12-40% and 2-29% gains in Figures 11-14 are measured end-to-end on A40/A800/H100 clusters, not computed from the DP recurrence, so the result is not equivalent to its inputs. The memory claim (maximum model size within 2.5% of Megatron) is also measured. The paper's self-citations (e.g., Ref. [12] pointing to the authors' own data repository for the pairwise overlap table, and Ref. [27] with a coauthor) are not load-bearing: the pairwise table is auxiliary data, and nnScaler is cited only as an example of hybrid parallelism. The DP additivity assumption for P(i,j) is a potential correctness/robustness limitation, but it is a modeling assumption, not a circular reduction; no equation or fitted parameter is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper contributes an engineering scheduler; there are no fitted physical constants or free parameters in the scientific sense. The load-bearing premises are the validity of the profiling-based cost model, the complementarity of activation memory, semantic preservation of the W-shape, and the negligible cost of the extra pipeline traffic.

assumptions (4)
  • domain assumption Pairwise operator-overlap costs P(i,j) measured in isolation are additive and context-independent during real training.
    The DP recurrence (Eq. 2) sums P(i,j) values as fixed constants; the paper does not test whether co-execution cost changes with memory pressure, prior kernels, or NCCL state.
  • domain assumption Forward and backward activation memory footprints of two strands are complementary triangles, so two micro-batches fit with under 3% extra memory.
    Central to the memory claim in Sec 3.3 and 6.5; verified only for one configuration (Llama-25B, 39B vs 40B limit), not as a general property.
  • domain assumption The W-shaped pipeline preserves the numerical semantics of training, so convergence and accuracy are unaffected.
    The paper asserts this in Sec 6.1 but provides no convergence curves or accuracy comparison.
  • domain assumption The W-shape's doubled pipeline Send/Recv traffic is negligible in common distributed LLM training.
    Stated in Sec 4.1 without quantification in the evaluated workloads.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution." pith.science (2026). https://pith.science/paper/FP53QKYR

@misc{pith2026241115871,
  author       = {Pith},
  title        = {Pith review of: Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FP53QKYR}},
  note         = {Machine review of arXiv:2411.15871}
}
read the original abstract

The growth of Large Language Models (LLMs) has necessitated large-scale distributed training. Highly optimized frameworks, however, still suffer significant losses in Model FLOPS utilization (often below 50%) due to large communication volumes. Meanwhile, our comprehensive profiling shows that the computation- and communication-intensive operators overlap well. This paper introduces DHelix, a novel micro-structure that dramatically improves the efficiency of LLM training inspired by the DNA structure. Central to DHelix's design is Strand Interleaving (SI), which views the continuous stream of training micro-batches through a GPU as two strands. DHelix juxtaposes the forward and backward passes of the two strands and performs a systematic optimization for an SI plan that co-schedules the operators from the opposite strands, enabled by operator-level overlap profiling results and a dynamic-programming based search algorithm. Meanwhile, DHelix enables the two strands to share model states and space for activation data, effectively accommodating two micro-batches with under 3% extra memory space. Dhelix seamlessly integrates with all forms of existing data/model parallelism, the most challenging being pipeline parallelism, thanks to its unique model folding design that results in a W-shaped pipeline. We evaluate DHelix training with the popular Llama and GPT dense models, plus the Phi Mixture of Expert (MoE) model, across 3 GPU clusters (A40, A800, and H100). Results show that it achieves 12-40% (up to 58% MFU) and 2-29% (up to 71% MFU) improvement on the 64-A40 and 64-A800 clusters, respectively, significantly outperforming state-of-the-art methods. On the H100 cluster, though the faster network reduces DHelix's profit margin, it makes cross-node tensor parallelism promising, a practice currently prohibitive due to communication costs.

Figures

Figures reproduced from arXiv: 2411.15871 by the authors.

Figure 1
Figure 1. Double-strand execution in DHelix on 4 GPUs Based on these results and inspired by the DNA struc￾ture, we propose DHelix (“Double-Helix LLM” training), a novel micro-structure that significantly improves the effi￾ciency of LLM training. The main idea in DHelix’s design is Strand Interleaving (SI), which views the continuous stream of training micro-batches through a GPU as two strands to be co-executed, 𝛼-strand and… view at source ↗
Figure 2
Figure 2. Sample result operator overlapping schedule by methods proposed in MegaScale [20], captured using the NVIDIA nsight profiling tool [42], in comparison to the exe￾cution follow achieved by Megatron-LM (top) and SP within each node, allowing their large amounts of inter-GPU communication to exploit the fast local-node net￾work connection, while scaling out with CP, PP, and EP to additional nodes. Collective Communicat… view at source ↗
Figure 3
Figure 3. gives the distribution of the total pre-training time among computation and three types of communication 8B 25B 39B 6.7B 18B 30B 31B 0.0 0.5 1.0 Ratio Llama GPT Phi TP Comm. CP Comm. EP Comm. Comp. others [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Sample memory allocation breakdown in training Llama-25B model, with 8192 sequence length and micro￾batch size 1, on 64 A40 GPUs with parallelism strategy of DP=8 and TP=8. • Both types of communication operators overlap well with both types of computation ones, though…
Figure 6
Figure 6. Figure 6: Pipeline scheduling with two strands: (a) V-shape from 1F1B and (b) Bi-direction While the idea has been explored by the Wavelet ap￾proach [50], its proposed Tick-Tock scheduling, which was used in data parallelism and considers the overlap between the peaks and valley…
Figure 8
Figure 8. Figure 8: Sample W-shaped pipeline schedule in DHelix The difference is that now the flow of the two strands across the GPUs are always heading the same way instead of opposite. The 𝛼-strand backward (green) and 𝛽-strand forward (blue) passes start from𝐺0, reach𝐺3, and then retu…
Figure 9
Figure 9. Figure 9: illustrates its overall workflow: (1) generating all possible operator sequences based on the appropriate DAG, (2) generating operator segments by partitioning a pair of for￾ward/backward operator sequences into contiguous pieces, and (3) using a dynamic programming al…
Figure 10
Figure 10. Figure 10: SI pairing results on different hardware. Here the dual-color blocks note computation/communication op￾erators in the backward pass that can be overlapped (no inter-dependency). Such “self-overlapping” operators within a strand is treated as a single operator in DHeli…
Figure 11
Figure 11. Figure 11: Overall performance w. Llama/GPT models. The tuple (x, y, z) under the bars gives DP, TP, and PP group sizes. and model sizes and achieves 27-40%, 15-28%, and 21-25% improvement compared to Megatron-LM, Intra-batch, and Wavelet+, respectively. GPT model training shows…
Figure 12
Figure 12. Figure 12: Llama/GPT training performance with long se￾quence. w in tuple (x, y, z, w) denotes CP group size. 6.2.2 Dense Models with Long Sequences. Considering the growing demand for long-context LLMs, we next evalu￾ate Llama and GPT model training with long sequences, dou￾bli…
Figure 14
Figure 14. Figure 14: Scaling model training with cross-node TP, w. transformer layer configuration from Llama3.1-405B [14]. For example, “L8 (TP8)” refers to a transformer model with 8 layers and a TP group size of 8. Wavelet+ shows slight improvement over Megatron-LM due to its simplisti…
Figure 16
Figure 16. Figure 16: Visualization of the single-strand and double-strand CUDA schedule of SI on our A800 cluster, based on traces from the experiment shown in [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 39 canonical work pages

  1. [51]

    Domino: Eliminating communication in llm training via generic tensor slicing and overlapping

    Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, and Olatunji Ruwase. Domino: Eliminating communication in llm training via generic tensor slicing and overlapping. arXiv preprint arXiv:2409.15241, 2024

    arXiv 2024

  2. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [2]

    Gradient compression supercharged high-performance data parallel dnn training

    Youhui Bai, Cheng Li, Quan Zhou, Jun Yi, Ping Gong, Feng Yan, Ruichuan Chen, and Yinlong Xu. Gradient compression supercharged high-performance data parallel dnn training. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles , pages 359–375, 2021

    work page 2021

  4. [3]

    Efficient all-to-all collective com- munication schedules for direct-connect topologies

    Prithwish Basu, Liangyu Zhao, Jason Fantl, Siddharth Pal, Arvind Krishnamurthy, and Joud Khoury. Efficient all-to-all collective com- munication schedules for direct-connect topologies. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, pages 28–41, 2024

    work page 2024

  5. [4]

    Chatbots and large language models in radiology: a practical primer for clinical and research applications

    Rajesh Bhayana. Chatbots and large language models in radiology: a practical primer for clinical and research applications. Radiology, 310(1):e232756, 2024

    work page 2024

  6. [5]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  7. [6]

    Flux: Fast software-based communication overlap on gpus through kernel fusion

    Liwen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Ziheng Jiang, Haibin Lin, et al. Flux: Fast software-based communication overlap on gpus through kernel fusion. arXiv preprint arXiv:2406.06858, 2024

    arXiv 2024

  8. [7]

    Centauri: Enabling efficient sched- uling for communication-computation overlap in large model train- ing via communication partitioning

    Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. Centauri: Enabling efficient sched- uling for communication-computation overlap in large model train- ing via communication partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Sys...

    2024

Show all 61 references
  1. [8]

    Optimizing large model training through overlapped activation recomputation

    Ping Chen, Wenjie Zhang, Shuibing He, Yingjie Gu, Zhuwei Peng, Kexin Huang, Xuan Zhan, Weijian Chen, Yi Zheng, Zhefeng Wang, et al. Optimizing large model training through overlapped activation recomputation. arXiv preprint arXiv:2406.08756, 2024

    2024 arXiv

  2. [9]

    Flashattention-2: Faster attention with better parallelism and work partitioning (2023)

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning (2023). arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  3. [10]

    Flashattention: Fast and memory-efficient exact attention with io- awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io- awareness. Advances in Neural Information Processing Systems , 35:16344–16359, 2022

    2022

  4. [11]

    [accessed-Oct-2024]

    deepseek. deepseek-hg. https://huggingface.co/deepseek-ai/ DeepSeek-V2.5, 2024. "[accessed-Oct-2024]"

    2024

  5. [12]

    [accessed- Aug-2024]

    dhelix. Overlap efficiency in h100. https://github.com/1926627357/ dhelix-data/blob/main/eff-overlap/eff_heatmap.pdf , 2024. "[accessed- Aug-2024]"

    2024

  6. [13]

    Liger: Interleaving intra- and inter- operator parallelism for distributed large model inference

    Jiangsu Du, Jinhui Wei, Jiazhi Jiang, Shenggan Cheng, Dan Huang, Zhiguang Chen, and Yutong Lu. Liger: Interleaving intra- and inter- operator parallelism for distributed large model inference. In Proceed- ings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice...

    2024

  7. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [15]

    Tic- tac: Accelerating distributed deep learning with communication sched- uling

    Sayed Hadi Hashemi, Sangeetha Abdu Jyothi, and Roy Campbell. Tic- tac: Accelerating distributed deep learning with communication sched- uling. Proceedings of Machine Learning and Systems , 1:418–430, 2019

    2019

  9. [16]

    Fastmoe: A fast mixture-of-expert training system

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. arXiv preprint arXiv:2103.13262, 2021

    2021 arXiv

  10. [17]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  11. [18]

    Breaking the computation and communication abstraction barrier in distributed machine learning workloads

    Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In Proceedings of th...

    2022

  12. [19]

    Priority-based parameter propagation for distributed dnn training

    Anand Jayarajan, Jinliang Wei, Garth Gibson, Alexandra Fedorova, and Gennady Pekhimenko. Priority-based parameter propagation for distributed dnn training. Proceedings of Machine Learning and Systems , 1:132–145, 2019

    2019

  13. [20]

    MegaScale: Scaling large language model training to more than 10,000 GPUs

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. MegaScale: Scaling large language model training to more than 10,000 GPUs. In 21st USENIX Symposium on Networked Systems Design and Implementation ...

    2024

  14. [21]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems , 5:341–353, 2023

    2023

  15. [22]

    Automatic horizontal fusion for gpu kernels

    Ao Li, Bojian Zheng, Gennady Pekhimenko, and Fan Long. Automatic horizontal fusion for gpu kernels. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) , pages 14–27. IEEE, 2022

    2022

  16. [23]

    Fold3d: Rethinking and parallelizing computational and communicational tasks in the training of large dnn models

    Fanxin Li, Shixiong Zhao, Yuhao Qing, Xusheng Chen, Xiuxian Guan, Sen Wang, Gong Zhang, and Heming Cui. Fold3d: Rethinking and parallelizing computational and communicational tasks in the training of large dnn models. IEEE Transactions on Parallel and Distributed Systems, 34(5...

    2023

  17. [24]

    Automated tensor model paral- lelism with overlapped communication for efficient foundation model training

    Shengwei Li, Zhiquan Lai, Yanqi Hao, Weijie Liu, Keshi Ge, Xiaoge Deng, Dongsheng Li, and Kai Lu. Automated tensor model paral- lelism with overlapped communication for efficient foundation model training. arXiv preprint arXiv:2305.16121, 2023

    2023 arXiv

  18. [25]

    Chimera: efficiently training large- scale neural networks with bidirectional pipelines

    Shigang Li and Torsten Hoefler. Chimera: efficiently training large- scale neural networks with bidirectional pipelines. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14, 2021

    2021

  19. [26]

    Zico: Efficient GPU memory sharing for concurrent DNN training

    Gangmuk Lim, Jeongseob Ahn, Wencong Xiao, Youngjin Kwon, and Myeongjae Jeon. Zico: Efficient GPU memory sharing for concurrent DNN training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 161–175, 2021

    2021

  20. [27]

    nnScaler:Constraint-Guided Parallelization Plan Generation for Deep Learning Training

    Zhiqi Lin, Youshan Miao, Quanlu Zhang, Fan Yang, Yi Zhu, Cheng Li, Saeed Maleki, Xu Cao, Ning Shang, Yilei Yang, et al. nnScaler:Constraint-Guided Parallelization Plan Generation for Deep Learning Training. In 18th USENIX Symposium on Operating Systems Design and Implementatio...

    2024

  21. [28]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  22. [29]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  23. [30]

    Libra: Contention-aware gpu thread allocation for data parallel training in high speed networks

    Yunzhuo Liu, Bo Jiang, Shizhen Zhao, Tao Lin, Xinbing Wang, and Chenghu Zhou. Libra: Contention-aware gpu thread allocation for data parallel training in high speed networks. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications , pages 1–10. IEEE, 2023

    2023

  24. [31]

    [accessed-Oct-2024]

    mistrialai. mixtral. https://huggingface.co/mistralai/Mixtral-8x7B- Instruct-v0.1, 2024. "[accessed-Oct-2024]". 14

    2024

  25. [32]

    Pipedream: Generalized pipeline parallelism for dnn train- ing

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn train- ing. In Proceedings of the 27th ACM symposium on operating systems principles,...

    2019

  26. [33]

    Efficient large- scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large- scale language model training on gpu clusters using megatron-lm. In Procee...

    2021

  27. [34]

    Sentence-t5: Scalable sen- tence encoders from pre-trained text-to-text models

    Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sen- tence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877, 2021

    2021 arXiv

  28. [35]

    [accessed-Oct-2024]

    NVDIA. H100 hardware configuration. https://resources.nvidia. com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet , 2024. "[accessed-Oct-2024]"

    2024

  29. [36]

    [accessed-Sept-2024]

    NVIDIA. Context parallelism overview. https://docs.nvidia.com/ megatron-core/developer-guide/latest/api-guide/context_parallel. html, 2024. "[accessed-Sept-2024]"

    2024

  30. [37]

    [accessed-Sept-2024]

    NVIDIA. Megatron-lm and megatron-core. https://github.com/ NVIDIA/Megatron-LM, 2024. "[accessed-Sept-2024]"

    2024

  31. [38]

    [accessed-Aug-2024]

    NVIDIA. Mps service. https://docs.nvidia.com/deploy/pdf/CUDA_ Multi_Process_Service_Overview.pdf,, 2024. "[accessed-Aug-2024]"

    2024

  32. [39]

    Nccl and mpi, 2024

    NVIDIA. Nccl and mpi, 2024. https://docs.nvidia.com/deeplearning/ nccl/user-guide/docs/mpi.html?highlight=alltoall#other-collectives- and-point-to-point-operations [Accessed: September 2024]

    2024

  33. [40]

    [accessed- Sept-2024]

    NVIDIA. Nccl operations. https://docs.nvidia.com/deeplearning/nccl/ user-guide/docs/usage/operations.html#allgather, 2024. "[accessed- Sept-2024]"

    2024

  34. [41]

    [accessed-Sept-2024]

    Nvidia. Nvidia connectx infiniband adapters. https://www.nvidia.com/ en-us/networking/infiniband-adapters/, 2024. "[accessed-Sept-2024]"

    2024

  35. [42]

    [accessed-Sept-2024]

    NVIDIA. Nvidia nsight systems. https://developer.nvidia.com/nsight- systems, 2024. "[accessed-Sept-2024]"

    2024

  36. [43]

    [accessed-Aug-2024]

    NVIDIA. Nvidia triton inference server boosts deep learning infer- ence. https://developer.nvidia.com/blog/nvidia-serves-deep-learning- inference, 2024. "[accessed-Aug-2024]"

    2024

  37. [44]

    A generic communication scheduler for distributed dnn training acceleration

    Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo. A generic communication scheduler for distributed dnn training acceleration. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 16–29, 2019

    2019

  38. [45]

    {Zero-offload}: Democratizing{billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. {Zero-offload}: Democratizing{billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564, 2021

    2021

  39. [46]

    Horovod: fast and easy dis- tributed deep learning in tensorflow

    Alexander Sergeev and Mike Del Balso. Horovod: fast and easy dis- tributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018

    2018 arXiv

  40. [47]

    Megatron-lm: Training multi- billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi- billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  41. [48]

    Optimus-cc: Efficient large nlp model training with 3d parallelism aware communication compression

    Jaeyong Song, Jinkyu Yim, Jaewon Jung, Hongsun Jang, Hyung-Jin Kim, Youngsok Kim, and Jinho Lee. Optimus-cc: Efficient large nlp model training with 3d parallelism aware communication compression. In Proceedings of the 28th ACM International Conference on Architectural Support...

    2023

  42. [49]

    Zero++: Extremely efficient collective communication for giant model training

    Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yux- iong He. Zero++: Extremely efficient collective communication for giant model training. arXiv preprint arXiv:2306.10209, 2023

    2023 arXiv

  43. [50]

    Wavelet: Efficient dnn training with tick-tock scheduling

    Guanhua Wang, Kehan Wang, Kenan Jiang, Xiangjun Li, and Ion Stoica. Wavelet: Efficient dnn training with tick-tock scheduling. Proceedings of Machine Learning and Systems , 3:696–710, 2021

    2021

  44. [52]

    Overlap communication with dependent computation via decomposition in large deep learning models

    Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, et al. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the ...

    2022

  45. [53]

    Egeria: Efficient dnn training with knowledge-guided layer freezing

    Yiding Wang, Decang Sun, Kai Chen, Fan Lai, and Mosharaf Chowd- hury. Egeria: Efficient dnn training with knowledge-guided layer freezing. In Proceedings of the Eighteenth European Conference on Computer Systems, pages 851–866, 2023

    2023

  46. [54]

    Hi-speed dnn training with espresso: Unleashing the full potential of gradient compression with near-optimal usage strategies

    Zhuang Wang, Haibin Lin, Yibo Zhu, and TS Eugene Ng. Hi-speed dnn training with espresso: Unleashing the full potential of gradient compression with near-optimal usage strategies. In Proceedings of the Eighteenth European Conference on Computer Systems , pages 867–882, 2023

    2023

  47. [55]

    Tacos: Topology- aware collective algorithm synthesizer for distributed machine learn- ing

    William Won, Midhilesh Elavazhagan, Sudarshan Srinivasan, Ajaya Durg, Samvit Kaul, Swati Gupta, and Tushar Krishna. Tacos: Topology- aware collective algorithm synthesizer for distributed machine learn- ing. arXiv preprint arXiv:2304.05301, 2023

    2023 arXiv

  48. [56]

    Gandiva: Introspective cluster scheduling for deep learning

    Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, et al. Gandiva: Introspective cluster scheduling for deep learning. In 13th USENIX Symposium on Operating Systems Design and Im...

    2018

  49. [57]

    Moe-infinity: Offloading-efficient moe model serving

    Leyang Xue. Moe-infinity: Offloading-efficient moe model serving. arXiv preprint arXiv:2401.14361, 2024

    2024 arXiv

  50. [58]

    Salus: Fine-grained gpu sharing primitives for deep learning applications

    Peifeng Yu and Mosharaf Chowdhury. Salus: Fine-grained gpu sharing primitives for deep learning applications. arXiv preprint arXiv:1902.04610, 2019

    1902 arXiv

  51. [59]

    Mics: near-linear scaling for train- ing gigantic model on public cloud

    Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. Mics: near-linear scaling for train- ing gigantic model on public cloud. arXiv preprint arXiv:2205.00119, 2022

    2022 arXiv

  52. [60]

    Alpa: Automating Inter-and Intra-Operator par- allelism for distributed deep learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating Inter-and Intra-Operator par- allelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems De...

    2022

  53. [61]

    On optimizing the communication of model parallelism

    Yonghao Zhuang, Lianmin Zheng, Zhuohan Li, Eric Xing, Qirong Ho, Joseph Gonzalez, Ion Stoica, Hao Zhang, and Hexu Zhao. On optimizing the communication of model parallelism. Proceedings of Machine Learning and Systems , 5, 2023. 15 G1 PA D4 A F1 PF2D3 R PF3W3F4 RPF4 PF3W4 A PF...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.