Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Automatically Planning Optimal Parallel Strategy for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An automatic planner selects the fastest parallel training strategy for transformer language models by simulating computation, communication, and overlap, and then pruning 99% of the search space.

desk verdict A useful extension of automatic parallel strategy search that treats micro-batch and global batch size as optimization variables, but the 96% accuracy claim is weakened by an undisclosed utilization model profiled on the test cluster itself. read the letter →

arxiv 2501.00254 v1 pith:YA27OMOW submitted 2024-12-31 cs.AI cs.CL

Zongbiao Li (1) , Xiezhao Li (1) , Yinghao Cui (1) , Yijun Chen (1) , Zhixuan Gu (1) , Yuxuan Liu (1) , Wenbo Zhu (1) , Fei Jia (1)
show 6 more authors
Ke Liu (1) Qifeng Li (1) Junyao Zhan (1) Jiangtao Zhou (1) Chenxi Zhang (1) Qike Liu (1) ((1) HUAWEI)
This is my paper · ORCID
classification cs.AIcs.CL
keywords automaticparallelstrategytrainingtimesimulation3Dparallelismmicrobatchsizeoptimizationsearchspacepruningtransformerlanguagemodelsthroughputestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the optimal parallel training strategy for a large transformer language model—how to split the model across data, tensor, and pipeline parallel dimensions, and what micro and global batch sizes to use—can be found automatically from model and hardware descriptions alone. To do this it builds a white-box simulation of training time, decoupling total time into computation, communication, and overlap, and reports that the simulation estimates actual training duration with an average accuracy of about 96%. The simulation is then used to prune the space of parallel configurations by roughly 99%, and the reduced space is enumerated to find the fastest strategy; in the paper's experiments the chosen strategy is always the globally optimal one. If this holds, practitioners would not need to run trial training jobs to choose parallel hyperparameters, avoiding significant wasted compute.

What carries the argument

The load-bearing object is the closed-form training-duration simulator of Equation 1, which decomposes a training epoch into computation, three kinds of communication, all-reduce computation, bubble time, and overlap. The decomposition is 'white-box' in the sense that the flow amounts are derived from the transformer architecture and the parallel degrees, while the hardware-dependent scaling factors—computing-power utilization ρ and the communication slow-down rate q—are treated as black-box profiling outputs. The second piece of machinery is the pruning logic: proving that total time has the form TT = (ϕ1 + ϕ2/G)St makes the global batch size monotone, and analyzing ∂TT/∂b shows an optimal micro batch size exists, together with memory boundary inequalities that restrict tensor parallelism t and pipeline parallelism p. Together these reduce the search space to a size that can be enumerated exactly.

What would settle it

Run the planning algorithm on a cluster or a transformer variant for which no profiling of computing-power utilization and bandwidth slow-down has been performed, and compare the predicted training time for each candidate strategy against actual measured times; the claim of 96% real-time accuracy and global optimality fails if the recommended strategy is not the fastest among a brute-force enumeration or the average estimation error exceeds a few percent.

Watch

Extended reading notes

Core claim

The central claim is that a training-duration simulation model, built by analyzing the transformer computation and communication at the operator level and calibrating only the hardware-dependent utilization and bandwidth factors through profiling, is accurate and order-preserving enough to support automated strategy search. The paper expresses total time per epoch as TT = (TF + TCT + TAT − TO + TCD + TAD + TB + TCP)·St/G, where TF is computation, TCT/TCP/TCD are communication for tensor, pipeline, and data parallelism, TAT/TAD are the corresponding all-reduce computations, TB is pipeline bubble time, TO is overlapped communication, and St/G is the number of steps. Using this model, the paper proves monotonicity with respect to global batch size, establishes that micro batch size has an interior optimum, and derives memory constraints that limit tensor and pipeline degrees. After these prunings—which reduce the feasible search space by approximately 99%—the algorithm enumerates the remaining strategies and returns the one with minimal simulated training time. Experiments on 16 NPUs across three model families show average estimation accuracy of 96–97% and that the recommended strategy is always the global optimum among those tested.

Load-bearing premise

The simulation depends on a black-box 'computing power utilization' factor and a bandwidth slow-down rate that are obtained by profiling the same cluster and model families on which the method is tested, so the reported 96% accuracy may reflect fitting rather than independent prediction.

Editorial extensions

If this is right

  • A user can plug in the model architecture and cluster specifications and receive the full parallel strategy (data, tensor, and pipeline degrees plus micro and global batch sizes) without running any trial training jobs.
  • Including micro batch size as a search variable is necessary: the experiments show the optimal micro batch size is an intermediate value (about 4) rather than 1 or the memory maximum, and overlooking it can miss the global optimum.
  • The monotonicity proof shows that larger global batch sizes reduce computation, bubble, and data-parallel communication time, but with diminishing returns; the planner treats it as a user-set trade-off against gradient randomness.
  • The memory boundary analysis shows that input activations cannot be evenly split by pipeline and tensor degrees, so tensor parallelism has a minimum requirement; this prunes configurations that would otherwise seem valid.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to replace the profiled computing-power utilization ρ with a predictive model based on operator shapes and hardware counters, which could allow the 96% accuracy to generalize to unseen clusters; the paper does not attempt this.
  • The same decoupling-and-prune pattern could be applied to sequence parallelism, expert parallelism for mixture-of-experts models, and optimizer-state parallelism, which the paper lists as future directions without implementing.
  • A stronger validation of the 'always globally optimal' claim would randomize model sizes, layer counts, and cluster interconnects rather than testing a fixed set of models on one cluster topology; the current evidence is consistent with a profile-fit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an automatic parallel strategy planner for transformer-based LLMs on a given hardware cluster. It models per-strategy training duration by decomposing time into computation, communication, overlap, and bubble terms (Eq. 1), uses this white-box model to prune the search space over tensor/pipeline/data parallelism degrees and micro/global batch sizes, and then enumerates the remaining strategies. Experiments on 16 Ascend 910b NPUs with three 7B-14B models report an average total-time estimation accuracy of about 96%, rank preservation among the top-five strategies, and the claim that the recommended strategy is always globally optimal.

Significance. If the simulation model were predictive for new models and clusters, the contribution would be practically valuable: it addresses a real pain point in LLM training by jointly optimizing multiple parallel degrees and batch sizes, and it provides explicit analytical formulas and memory-bound reasoning rather than a pure black-box search. The paper also deserves credit for including micro and global batch sizes in the optimization, which comparable tools often fix as constants. However, the central predictive claim rests on an undisclosed profiled utilization model and on validation that is largely in-sample, so the significance is conditional on closing that gap.

major comments (4)
  1. [Training Time Estimation, Computation duration; Eq. (2)-(3)] The 96% accuracy claim is not independently testable because the computation-time estimate is scaled by a computing-power utilization ρ(b,s,h,t) whose functional form and fitting procedure are not given; the text says 'The specific modeling process is not given in the paper.' This ρ is profiled on the same cluster and the same model families that are later used for Table 2 validation, so the reported average accuracy may reflect in-sample fitting rather than predictive performance. To support the abstract's promise of planning from model and hardware information alone, the paper must either disclose the ρ model or validate on held-out models/clusters that were not used for profiling.
  2. [Training Time Estimation, Pipeline parallelism; Eq. (8)] Equation (8) defines bubble time as TBP = (TT + TCT + TART − TO) × (p − 1)/m, but TT in Eq. (1) already includes the bubble term TB. As written, the model is circular unless TT on the right-hand side denotes a different quantity, such as per-micro-batch time excluding bubble, which is not stated. This ambiguity makes the objective function in Eq. (13) not well-defined and must be resolved before the pruning analysis can be trusted.
  3. [Strategy Searching, Search space pruning; Eq. (16)] The proof of the optimal finite micro-batch size b is unsupported. The paper claims that 'since ρ is inversely proportional to b, the sign of the partial derivative in the right expression of Formula 16 is negative. Therefore, there exists an optimal value for b.' If ρ ∝ 1/b, the derivative in Eq. (16) contains both negative terms from ∂ρ/∂b and positive terms ω3ρ + ω4, so its sign is not uniformly negative; a finite optimum would require showing the derivative crosses from negative to positive. Moreover, because ρ's functional form is undisclosed, the monotonicity assumption cannot be independently verified. Since pruning b is essential to making enumeration tractable, this argument needs either a corrected proof or an empirical justification across models and batch sizes.
  4. [Experiment and Result, Rank preservation; Table 2] The claim that the recommended strategy is 'always globally optimal' exceeds the evidence presented. Table 2 profiles only five strategies per model, which are apparently selected using the model's own ranking; there is no comparison against the full pruned search space or against an exhaustive baseline. The low communication accuracy (as low as 73.63% for Qwen-14b) and overlap accuracy (as low as 58.42% for Aquila2-7b) leave substantial room for ranking errors among near-optimal strategies. The paper should either validate against a larger set of strategies, including those outside the model's top five, or temper the global-optimality claim accordingly.
minor comments (5)
  1. [Notation] Notation is inconsistent: Table 1 lists the bubble time as TB, but Eq. (8) writes TBP; Eq. (8) uses TART while Table 1 and Eq. (1) use TAT for the tensor-parallel all-reduce computation time. Please unify these symbols.
  2. [Eq. (14) and Algorithm 1] Equation (14) is ambiguous as printed: 'G = bmd / pt × MN P U≥ Mm' should be parenthesized, and the memory constraint is written as a chained inequality that mixes G and the memory condition. Algorithm 1 uses the variable N in the loop bound without defining it; presumably it is the maximum allowed pipeline degree, but this should be stated explicitly.
  3. [Table 1] In Table 1, the row for ρ reads 'Computing power utilization (Re)'; the 'Re' appears to be an artifact or a typo and should be removed.
  4. [Abstract and Conclusion] The abstract states that the algorithm 'can estimate the parallel training duration in real time with an average accuracy of 96%' and that the recommended strategy is 'always globally optimal'; these claims are repeated in the conclusion. Given the validation limitations, the conclusion should state the conditions under which these claims hold, such as the cluster and model configurations tested.
  5. [Related Work] There is a typo in the Related Work section: 'Activetion are communicated point-to-point' should be 'Activation'.

Circularity Check

2 steps flagged · score 6.0 of 10

The 96% duration accuracy is a fit: the dominant computation term uses an undisclosed ρ profiled on the same cluster/models, so the 'prediction' reduces to the profiling inputs.

  1. fitted input called prediction [Training Time Estimation, Computation duration (Eq. 2); Auto Parallelism Process; Table 2]
    "The uncertain part includes the utilization rate of computing power and effective transmission bandwidth of the cluster, which are dynamic during the training process and obtained through profiling. ... The computing power utilization of the NPU is related to the operator implementation and hardware. Here, we treat it as a black box and modelling it by small-scale profiling. Specifically, computing power utilization ρ is considered as a function of b, s, h, t. ... The specific modeling process is not given in the paper"

    The duration estimate that the abstract calls a prediction is TT in Eq. 1, whose dominant term TF in Eq. 2 is TF=(η1+η2p)Gρ. ρ is not derived from model/hardware parameters; it is 'obtained through profiling' on the same 16-NPU Ascend cluster and the same models that Table 2 uses as ground truth. The disclosed equations therefore reduce the '96% accuracy' and 'global optimal strategy' results to the quality of an undisclosed fit of ρ (plus profiled q and an unspecified all-reduce coefficient). Nothing in the paper holds out a cluster or model not used for profiling, so the accuracy claim is a statement about fitting, not an independent prediction.

  2. fitted input called prediction [Training Time Estimation, Communication duration (Eq. 6 and all-reduce coefficient)]
    "We evaluate this effect using a slow rate q, q obtained by profiling. ... This part of the calculation is not provided in the paper, but can be obtained by multiplying the corresponding communication time by a computing power/bandwidth coefficient."

    The communication terms also depend on profiled or undisclosed coefficients: Eq. 6 scales TP communication by q obtained by profiling, and the all-reduce computation terms TAT/TAD are defined only as a proportion of communication time with an unspecified coefficient. These fitted values are inputs to the same TT that is compared with real duration in Table 2, so the reported communication and total accuracy partly measure these fits rather than validating the analytic model.

full rationale

The paper's FLOP/communication-volume bookkeeping (Eqs. 2–12) is a self-contained analytic model and is not circular by itself. The circularity enters at the point where the paper converts this bookkeeping into a numerical duration estimate: the hardware-efficiency factors ρ and q, and an all-reduce coefficient, are treated as black boxes 'obtained through profiling,' with ρ's 'specific modeling process ... not given.' Table 2 then validates the estimated TT on the same 16-NPU Ascend cluster and the same models (Baichuan2-7b, Qwen-14b, Aquila2-7b) that would supply that profiling. Because ρ and q can be chosen to absorb the difference between the analytic FLOP formula and the measured times, the reported 96–97% accuracy is a measure of the fitted profile, not an independent prediction of training duration for new models or clusters. There is no evidence of load-bearing self-citation: the cited prior work (InternEvo, Megatron, Galvatron, Calculon) is not invoked to define ρ or to force the optimal-strategy conclusion. The pruning proofs (e.g., monotonicity in G, optimal b) are derived from the stated equations and are independent. Score 6 reflects that the central 'estimate in real time with 96% accuracy' claim partially reduces to fitted inputs; had the validation used a held-out model or cluster, the score would be near 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on profiled quantities (ρ, q) and an unstated coefficient, plus several domain assumptions about hardware topology and pipeline scheduling. No new physical or conceptual entities are invented.

free parameters (3)
  • computing power utilization ρ = not disclosed (function of b, s, h, t)
    Used in Eq. 2 and Eq. 12 to convert FLOPs into time. Obtained through profiling; 'the specific modeling process is not given in the paper'. Dominates the total time estimate.
  • communication slow-down rate q = not disclosed
    Used in TP communication duration, Eqs. 6-7, to model bandwidth degradation as communication members increase. Obtained through profiling.
  • all-reduce compute duration coefficient = not disclosed
    The paper states TAT (and TAD) are 'not provided in the paper, but can be obtained by multiplying the corresponding communication time by a computing power/bandwidth coefficient.' This coefficient is a free proportionality constant.
assumptions (5)
  • standard math Transformer compute is dominated by tensor multiplications with the FLOP counts in Eq. 3
    The simulator assumes these formulas match the actual operator mix on Ascend NPUs.
  • domain assumption PipeDream-Flush (1F1B) scheduling is used, giving the bubble formula in Eq. 8
    The choice of pipeline schedule directly determines bubble time; the paper models only this schedule.
  • domain assumption Within-server mesh topology and between-server ring topology determine communication costs, with a slowdown factor q when group size grows
    Used in Eqs. 4-7; another topology would change the communication duration formulas.
  • ad hoc to paper ρ is inversely proportional to b (asserted in the b-pruning argument)
    The sign analysis of Eq. 16 depends on this monotonicity claim, which is stated without measurement or derivation.
  • domain assumption Activation memory is not tensor-parallelized and 1F1B reduces activation from bm to bp
    Memory constraints in Eqs. 17-20 rely on this to bound p and t.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatically Planning Optimal Parallel Strategy for Large Language Models." pith.science (2026). https://pith.science/paper/YA27OMOW

@misc{pith2026250100254,
  author       = {Pith},
  title        = {Pith review of: Automatically Planning Optimal Parallel Strategy for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YA27OMOW}},
  note         = {Machine review of arXiv:2501.00254}
}
read the original abstract

The number of parameters in large-scale language models based on transformers is gradually increasing, and the scale of computing clusters is also growing. The technology of quickly mobilizing large amounts of computing resources for parallel computing is becoming increasingly important. In this paper, we propose an automatic parallel algorithm that automatically plans the parallel strategy with maximum throughput based on model and hardware information. By decoupling the training time into computation, communication, and overlap, we established a training duration simulation model. Based on this simulation model, we prune the parallel solution space to shorten the search time required. The multi-node experiment results show that the algorithm can estimate the parallel training duration in real time with an average accuracy of 96%. In our test, the recommendation strategy provided by the algorithm is always globally optimal.

Figures

Figures reproduced from arXiv: 2501.00254 by the authors.

Figure 1
Figure 1. The overall workflow of our automatic parallel al [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. shows the stages of two all-reduce operations, with the direction of the arrows indicating forward propaga￾tion. During backward propagation, the communication be￾tween X1 and X2 can be synchronized with the computa￾tions and updates of WA1, WA2, WP 1, and WP 2. (a) All-reduce of multi head attention (b) All-reduce of feed-forward [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. The impact of micro batch size on throughput. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: The impact of global batch size on throughput. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 16 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ascend. 2024 a . MindSpeed. https://gitee.com/ascend/MindSpeed

    work page 2024

  4. [4]

    Ascend. 2024 b . ModelLink. https://gitee.com/ascend/ModelLink

    work page 2024

  5. [5]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  6. [6]

    Chen, Q.; Gu, D.; Wang, G.; Chen, X.; Xiong, Y.; Huang, T.; Hu, Q.; Jin, X.; Wen, Y.; Zhang, T.; et al. 2024. Internevo: Efficient long-sequence large language model training via hybrid parallelism and redundant sharding. arXiv preprint arXiv:2401.09149

    arXiv 2024

  7. [7]

    Dean, J.; Corrado, G.; Monga, R.; Chen, K.; Devin, M.; Mao, M.; Ranzato, M.; Senior, A.; Tucker, P.; Yang, K.; et al. 2012. Large scale distributed deep networks. Advances in neural information processing systems, 25

    work page 2012

  8. [8]

    Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; Casas, D. d. L.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    arXiv 2022

Show all 29 references
  1. [9]

    V.; Wu, Y.; et al

    Huang, Y.; Cheng, Y.; Bapna, A.; Firat, O.; Chen, D.; Chen, M.; Lee, H.; Ngiam, J.; Le, Q. V.; Wu, Y.; et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32

    2019

  2. [10]

    Isaev, M.; McDonald, N.; Dennison, L.; and Vuduc, R. 2023. Calculon: a methodology and tool for high-level co-design of systems and large language models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1--14

    2023

  3. [11]

    Jia, Z.; Zaharia, M.; and Aiken, A. 2019. Beyond data and model parallelism for deep neural networks. Proceedings of Machine Learning and Systems, 1: 1--13

    2019

  4. [12]

    B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D

    Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  5. [13]

    J.; Awan, A

    Kim, Y. J.; Awan, A. A.; Muzio, A.; Salinas, A. F. C.; Lu, L.; Hendy, A.; Rajbhandari, S.; He, Y.; and Awadalla, H. H. 2021. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.10465

    2021 arXiv

  6. [14]

    A.; Casper, J.; Lym, S.; McAfee, L.; Andersch, M.; Shoeybi, M.; and Catanzaro, B

    Korthikanti, V. A.; Casper, J.; Lym, S.; McAfee, L.; Andersch, M.; Shoeybi, M.; and Catanzaro, B. 2023. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 341--353

    2023

  7. [15]

    G.; Park, J

    Li, M.; Andersen, D. G.; Park, J. W.; Smola, A. J.; Ahmed, A.; Josifovski, V.; Long, J.; Shekita, E. J.; and Su, B.-Y. 2014 a . Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on operating systems design and implementation (OSDI 14), 583--598

    2014

  8. [16]

    G.; Smola, A

    Li, M.; Andersen, D. G.; Smola, A. J.; and Yu, K. 2014 b . Communication efficient distributed machine learning with the parameter server. Advances in Neural Information Processing Systems, 27

    2014

  9. [17]

    Li, S.; Xue, F.; Baranwal, C.; Li, Y.; and You, Y. 2021. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120

    2021 arXiv

  10. [18]

    Miao, X.; Wang, Y.; Jiang, Y.; Shi, C.; Nie, X.; Zhang, H.; and Cui, B. 2022. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism. Proceedings of the VLDB Endowment, 16(3): 470--479

    2022

  11. [19]

    Narayanan, D.; Phanishayee, A.; Shi, K.; Chen, X.; and Zaharia, M. 2021 a . Memory-efficient pipeline-parallel dnn training. In International Conference on Machine Learning, 7937--7947. PMLR

    2021

  12. [20]

    Narayanan, D.; Shoeybi, M.; Casper, J.; LeGresley, P.; Patwary, M.; Korthikanti, V.; Vainbrand, D.; Kashinkunti, P.; Bernauer, J.; Catanzaro, B.; Phanishayee, A.; and Zaharia, M. 2021 b . Efficient Large-Scale Language Model Training on GPU Clusters. CoRR, abs/2104.04473

    2021 arXiv

  13. [21]

    Qi, P.; Wan, X.; Huang, G.; and Lin, M. 2023. Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241

    2023 arXiv

  14. [22]

    Rajbhandari, S.; Ruwase, O.; Rasley, J.; Smith, S.; and He, Y. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the international conference for high performance computing, networking, storage and analysis, 1--14

    2021

  15. [23]

    Rashidi, S.; Denton, M.; Sridharan, S.; Srinivasan, S.; Suresh, A.; Nie, J.; and Krishna, T. 2021. Enabling compute-communication overlap in distributed deep learning training platforms. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), 540-...

    2021

  16. [24]

    Rasley, J.; Rajbhandari, S.; Ruwase, O.; and He, Y. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 3505--3506

    2020

  17. [25]

    Shoeybi, M.; Patwary, M.; Puri, R.; LeGresley, P.; Casper, J.; and Catanzaro, B. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053

    2019 arXiv

  18. [26]

    S.; Maggioni, M.; Zhang, Q.; et al

    Wang, S.; Wei, J.; Sabne, A.; Davis, A.; Ilbeyi, B.; Hechtman, B.; Chen, D.; Murthy, K. S.; Maggioni, M.; Zhang, Q.; et al. 2022. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conf...

    2022

  19. [27]

    Yang, A.; Xiao, B.; Wang, B.; Zhang, B.; Bian, C.; Yin, C.; Lv, C.; Pan, D.; Wang, D.; Yan, D.; et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

    2023 arXiv

  20. [28]

    Yang, B.; Zhang, J.; Li, J.; R \'e , C.; Aberger, C.; and De Sa, C. 2021. Pipemare: Asynchronous pipeline parallel dnn training. Proceedings of Machine Learning and Systems, 3: 269--296

    2021

  21. [29]

    Zhang, B.-W.; Wang, L.; Li, J.; Gu, S.; Wu, X.; Zhang, Z.; Gao, B.; Ao, Y.; and Liu, G. 2024. Aquila2 Technical Report. arXiv preprint arXiv:2408.07410

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.