Pith. sign in

REVIEW 4 major objections 6 minor 48 references

Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simulator-guided branch-and-bound search can cut LLM training time on mixed-GPU clusters by up to 4.69x while also adapting to shifting network bandwidth.

desk verdict A plausible multi-edge link abstraction and an honest limitations section, but the simulation-only evidence and weak baseline don't yet support the 4.69x claim. read the letter →

arxiv 2506.02787 v1 pith:3V24BM6L submitted 2025-06-03 cs.DC cs.AI

Ruilong Wu , Xinjiao Li , Yisu Wang , Xinyu Chen , Dirk Kutscher This is my paper
classification cs.DCcs.AI
keywords dynamicnetworkshybridparallelismdistributedtrainingheterogeneouscomputingautomaticparallelizationbranch-and-boundsearchperformancesimulationoperatorsplitting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that automatic parallelization for large-model training should account for heterogeneous GPU speeds and dynamic network bandwidth together, and that a search over operator-level parallel strategies can exploit both. Its planner models each device-to-device connection as multiple physical links with distinct bandwidths, uses a simulator to predict operator and communication times, and prunes infeasible strategies during a branch-and-bound search. If the claim holds, LLM training on cloud clusters with mixed GPU generations and fluctuating bandwidth could automatically find workload splits that cut training time by up to 4.69x relative to an equal-workload baseline without manual tuning.

What carries the argument

The central machinery is a parallelized branch-and-bound search whose objective values come from a CUDA-kernel simulator. Two abstractions carry the argument: a multi-edge link model, which represents each device-to-device connection as several physical links with separate bandwidth and conflict states instead of one average bandwidth, and an operator-splitting and recombination step, which creates new candidate operators (such as reduce-scatter plus all-gather in place of all-reduce) that can be mapped onto heterogeneous devices. The branch-and-bound maintains a priority queue of partial operator assignments, discards branches whose lower-bound estimate already exceeds the best upper bound, and evaluates surviving candidates concurrently in the simulator. Data-dependency, per-device memory, and per-link bandwidth constraints are enforced while minimizing the weighted sum of model completion times.

What would settle it

A concrete check would be to throttle bandwidth on a real cluster containing fast and slow GPUs, run the planner's chosen strategy for one training epoch, and compare wall-clock time against the equal-workload baseline; if the planner is not faster, the 4.69x claim is a simulator artifact. A more direct test measures the simulator's predicted kernel and communication times against timers on the same hardware.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that automatic parallel planning for large-model training should minimize the weighted makespan of operator graphs over a device graph in which each physical link is represented by multiple edges with independent bandwidth and contention states. The paper argues that execution and communication times in heterogeneous, dynamically changing clusters cannot be captured by linear roofline-style models, so it uses a simulator to predict operator times and to evaluate many candidate schedules. It then searches operator splitting, fusion, device assignment, and communication-link choices with a branch-and-bound procedure that prunes any candidate whose estimated cost cannot beat the best known solution. The reported result is that this finds faster schedules than an equal-workload baseline: up to 4.69x faster one-epoch training on a strongly heterogeneous GPU mix, marginal gains on a similar GPU mix, and up to 52% lower execution time under low bandwidth by avoiding oversized tensor-parallel groups.

Load-bearing premise

The load-bearing premise is that the simulator's predicted execution times faithfully represent real heterogeneous GPUs and real network conditions; the paper's speedups are calculated from those predictions, not from measured training runs.

Editorial extensions

If this is right

  • On clusters mixing recent and older GPUs, automatically splitting operators and assigning sub-operators to devices can shorten one-epoch training time by up to 4.69x relative to an equal-workload baseline.
  • Under low available bandwidth, the search prefers smaller tensor-parallel groups for smaller models, cutting execution time by up to 52% compared with the previously selected strategy.
  • Under stable, similar-performance conditions, the method remains competitive, producing speedups of roughly 1.01 to 1.03x rather than regressing.
  • Because the search uses simulated execution times, it can capture nonlinear effects of operator fusion and kernel-level device behavior that closed-form cost models miss.
  • Pruning and parallel simulation make the exponential search over operator splits and device mappings feasible enough for preliminary use, with CPU-side search speed as the remaining bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the simulator's kernel predictions transfer faithfully to real hardware, the same search could be pushed to lower memory levels (shared memory and registers), a direction the paper names but does not implement.
  • A direct real-cluster benchmark with throttled bandwidth would convert the reported speedups from simulation-based projections into measured numbers; until then the 4.69x figure is an upper bound.
  • The weighted-makespan objective and multi-edge device graph could be adapted to heterogeneous cloud settings where links span wide-area networks and carry different monetary costs, making the planner optimize cost rather than time.
  • The branch-and-bound structure does not depend on the simulator specifically: substituting a learned runtime predictor would trade accuracy for search speed and allow larger models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an automatic parallelization framework for distributed LLM training on heterogeneous clusters with dynamic network conditions. It introduces a multi-edge physical link abstraction to model unequal and conflicting interconnects, formulates operator-level scheduling as a constrained makespan minimization problem, and proposes a parallelized branch-and-bound search whose cost estimates come from the SimAI simulator. The evaluation uses SimAI to compare the proposed method against Megatron's default configuration on four LLM sizes under heterogeneous-GPU and low/high-bandwidth settings, reporting speedups up to 4.69x on RTX4090D/V100 clusters and up to 52% execution-time reductions when switching to newly selected strategies under low bandwidth. The paper explicitly acknowledges that the current implementation is limited to Megatron-LM-style layer-level parallelism and that the search space grows exponentially.

Significance. If validated on real hardware, the approach would be a useful step toward automatic parallelization in realistic heterogeneous and dynamic clusters, and the multi-edge abstraction is a plausible improvement over single-bandwidth link models. A strength of the paper is that it uses SimAI, an external simulator, rather than fitting constants to the reported results, and it includes a candid limitations section. However, the significance is conditional: the current evidence base consists entirely of simulator predictions, the baseline is only Megatron's default configuration rather than the state-of-the-art planners cited in the introduction, and the dynamic-network experiments compare two static bandwidth values rather than exercising topology changes. The claimed fine-grained operator-splitting mechanism is also not what the experiments actually evaluate.

major comments (4)
  1. [Section 4, Figure 6, Section 6] The central speedup claim rests entirely on SimAI predictions, and the same simulator provides both the search objective (Section 3.3) and the evaluation metric (Figure 6). Section 4 states that 'we utilize SimAI to simulate task execution times', and Figure 6 reports these simulated executions as Megatron versus 'Our Method'. Because no validation of SimAI against real hardware measurements is provided, and no error bars or sensitivity analysis are reported, the 'up to 4.69 times' speedup in the Conclusion is a self-prediction of the simulator rather than a demonstrated training improvement. The risk is particularly acute in the low-bandwidth and highly heterogeneous regimes, where small errors in predicted communication cost or device throughput could produce large relative speedups.
  2. [Section 4, Models and Baselines; Abstract] The evaluation compares only against 'Megatron using its default configuration', yet the Introduction cites ALPA, AMP, Metis, and Galvatron as existing automatic search frameworks, and the Abstract claims the method achieves 'performance competitive with state-of-the-art methods'. No experiments against these planners or other heterogeneity-aware baselines are reported. Without such comparisons, the abstract claim is unsupported, and it is unclear whether the proposed search improves on existing automatic parallel planners or only on a single default configuration.
  3. [Section 4.2, Figure 6(c)] The dynamic-network scenario does not actually evaluate a dynamic network. Section 2.2 defines dynamic networks as temporal graphs with a time-dependent edge set E(t), including bandwidth fluctuations and node failures, but Section 4.2 compares only two static bandwidth values, 1GB and 100GB, using a fixed set of V100 GPUs. No experiment varies bandwidth over time, changes topology, removes nodes, or introduces failures. Consequently, the claim of 'improved adaptability in complex, dynamic scenarios' is not demonstrated by the reported experiments.
  4. [Section 5, first limitation] The paper's own limitation statement says the implementation is 'limited to parallel strategies defined by the Megatron-LM' and relies on 'coarse-grained model-level task assignments' rather than native operator-level splitting. Yet the Introduction, Section 3, and the Conclusion present fine-grained operator splitting and recombination as the core mechanism behind the speedups. The experiments in Figure 6 therefore exercise layer-level assignment only, so the reported speedups do not validate the proposed operator-splitting mechanism. The central claim and the evaluated artifact are mismatched.
minor comments (6)
  1. [Figure 6(c)] The x-axis and y-axis of Figure 6(c) are labeled 'Model Size' and 'Relative Execution Time', but the text mentions cluster sizes of 8, 16, 64, and 256 GPUs without indicating how those are represented in the figure; the caption should specify the cluster configuration for each bar.
  2. [Section 2.1, Equations (1)-(2)] The notation in Equation (1) is confusing: 'roofline_BW' is written as a variable, and the units of FLOPs_p (peak FLOP/s) and K (FLOPs per access) are mixed inside the min operation. A cleaner formulation would separate peak throughput and memory-bound throughput.
  3. [Algorithm 1] Algorithm 1 is titled 'Parallel Branch-and-Bound Search', but the pseudocode contains no explicit parallel constructs; the body of the while loop processes one node at a time. The parallelism described in Section 3.4 as 'multi-threading' should either be reflected in the algorithm or the naming should be clarified.
  4. [Reference [37]] The SimAI reference lacks a publication venue, year, and page or arXiv identifier, which makes it difficult for readers to verify the simulator's capabilities and assumptions.
  5. [Section 6, Conclusion] The sentence 'newly identified parallel strategies can reduce execution times by up to 52% compared with previously optimal strategies' is misleading: the 52% figure comes from comparing the same strategy under 1GB versus 100GB static bandwidth, not from a dynamic change in the network. The wording should distinguish a static low-bandwidth penalty from an adaptation benefit.
  6. [Section 4, Models and Baselines] The description of 'Megatron using its default configuration' is underspecified; the paper should state which tensor/pipeline/data parallelism degrees and which communication settings were used as the baseline, since these choices directly affect the reported speedups.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the planner optimizes a SimAI-predicted cost that is also the reported metric, but SimAI is an external simulator and no parameters are fitted to the results, so the speedups are search outputs rather than constructional tautologies.

full rationale

The derivation chain is a standard simulator-in-the-loop search: Section 4 states 'we utilize SimAI to simulate task execution times ... to guide the parameter optimization,' and Figure 6 reports those same simulated execution times, which the Conclusion converts into 'speedups of up to 4.69 times.' One might worry that using the same simulator as both search objective and evaluation metric makes the reported improvement tautological. It does not: SimAI is an external, cited performance model ([37]) with no stated constants fitted to the paper's results, and the branch-and-bound search (Section 3.3) produces plans as outputs, so a lower simulated makespan is a genuine optimization outcome rather than an identity. The speedup's validity therefore rests on SimAI's fidelity to real GPUs and networks, which the paper does not validate; that is an evidence-quality limitation, not circular reasoning. Section 5 candidly limits the implementation to Megatron-LM strategies, so the operator-level splitting claims are not exercised by the evaluation, but this narrows the demonstration rather than making the derivation self-referential. No load-bearing self-citations or definitional reductions were found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central claims rest on several assumptions that the paper does not independently validate. Notably, the simulator's accuracy is the linchpin: the search optimizes the simulator's cost model, and the evaluation measures the same simulator's outputs, so the speedup numbers are only as trustworthy as SimAI. The multi-edge abstraction, the deterministic time model, and the admissibility of the branch-and-bound pruning are all asserted rather than demonstrated. There are no fitted scalar parameters in the traditional sense, but the search is driven by a black-box cost function F(N) whose internals are not specified. These points collectively mean the reader pays for the core mechanism from upstream assumptions, not from demonstrated measurements.

assumptions (5)
  • domain assumption The simulator (SimAI) provides sufficiently accurate execution and communication time predictions for all devices and network conditions.
    Section 4 states the evaluation 'utilizes SimAI to simulate task execution times' and uses these as the basis for both search and comparison; no validation against real hardware is given.
  • domain assumption Multiple physical links between devices should be modeled as concurrent or conflicting resources (multi-edge assumption).
    Introduced in Section 3.1 with DGX H100 and TPU examples, but no experiment isolates its effect on the reported results.
  • domain assumption Operator execution and communication times are deterministic, as formulated in Section 3.2.
    The introduction describes execution time uncertainty, but the optimization model uses deterministic estimates, ignoring the variance it motivates.
  • domain assumption The branch-and-bound search can prune using cost estimates F(N) without losing the optimal solution.
    Algorithm 1 prunes nodes when F(N_child) >= best_UB; the paper does not prove F is an admissible lower bound, so the optimality guarantee is assumed.
  • domain assumption The Megatron-LM strategy space is representative enough to demonstrate the method's benefits.
    Section 5 admits the evaluation is limited to Megatron-LM defined parallel strategies, so the operator-level splitting design is not actually tested.
invented entities (1)
  • multi-edge physical link abstraction
    purpose: To model the multiple physical links between devices (e.g., NVLink, PCIe, NVSwitch paths) as separate resources with individual bandwidth and contention states, rather than a single aggregated link.
    The paper introduces this as a core modeling contribution in Section 3.1 but provides no experiment demonstrating its predictive benefit; the evaluation uses coarse layer-level Megatron strategies, so the abstraction is not directly exercised.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization." pith.science (2026). https://pith.science/paper/3V24BM6L

@misc{pith2026250602787,
  author       = {Pith},
  title        = {Pith review of: Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3V24BM6L}},
  note         = {Machine review of arXiv:2506.02787}
}
read the original abstract

Hybrid parallelism techniques are essential for efficiently training large language models (LLMs). Nevertheless, current automatic parallel planning frameworks often overlook the simultaneous consideration of node heterogeneity and dynamic network topology changes, limiting their effectiveness in practical applications. In this paper, we address these limitations by modeling heterogeneous nodes within dynamically changing network environments and leveraging simulation-based strategies to determine optimal parallel configurations. Our approach enables fine-grained workload allocation tailored for heterogeneous nodes and complex network scenarios, achieving performance competitive with state-of-the-art methods under regular and stable network conditions. Additionally, we introduce a strategy pruning technique to rapidly discard infeasible parallel configurations, substantially reducing the search space and accelerating the search process through parallel execution within the simulator. Preliminary evaluations confirm that our method notably enhances training performance on heterogeneous nodes and demonstrates improved adaptability in complex, dynamic scenarios such as cloud computing environments.

Figures

Figures reproduced from arXiv: 2506.02787 by the authors.

Figure 2
Figure 2. Attention Throughput: H100 vs V100 as the number of floating-point operations per memory access, computed as: 𝐾 = FLOPs𝑘 mem𝑘 (2) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Representative Scenarios on hybrid parallelism: S1 Dynamic Bandwidth Variations; S2 Heterogeneous GPU Per￾formance; S3 Device changes and connection adjustments caused by node failures 2.1 Performance Heterogeneity Performance heterogeneity refers to variations in computa￾tional speed and capabilities among devices of the same type. Even if all nodes within a cluster employ GPUs that share the same instruction set (… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Hierarchy of GPU memory bandwidth optimization levels. The first level represents inter-device connections, providing a bandwidth ranging from several GB/s to tens of GB/s. The second level indicates global memory, with a bandwidth typically ranging from hundreds of GB…
Figure 5
Figure 5. Figure 5: Two typical types of links: (a) unequal bandwidth, (b) conflicting connections. 3.2 Problem Formulation We formulate an operator-level scheduling and resource alloca￾tion problem for distributed DNN tasks across a heterogeneous multi-edge device graph. The goal is to o…
Figure 6
Figure 6. Figure 6: Comparison of execution times for training one epoch under heterogeneous computing devices and dynamic network conditions: (a) RTX4090D combined with L20 GPUs, (b) RTX4090D combined with V100 GPUs, and (c) relative execution times for different tensor parallelism (TP) …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 28 canonical work pages

  1. [1]

    Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heteroge- neous distributed systems. arXiv preprint arXiv:1603.04467 (2016)

    arXiv 2016

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Ka- plan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Ad- vances in neural information processing systems 33 (2020), 1877–1901

    work page 2020

  3. [3]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to- End Optimizing Compiler for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . USENIX As- sociation, Ca...

    work page 2018

  4. [4]

    NVIDIA Corporation. 2017. NVIDIA DGX-1 System Architecture Whitepaper. https://www.azken.com/images/dgx1_images/dgx1-system- architecture-whitepaper1.pdf . Accessed: 2025-03-10

    work page 2017

  5. [5]

    NVIDIA Corporation. 2017. NVIDIA DGX-1 with Tesla V100 System Architecture. https://images.nvidia.com/content/pdf/dgx1-v100-system- architecture-whitepaper.pdf. Accessed: 2025-03-10

    work page 2017

  6. [6]

    NVIDIA Corporation. 2020. NVIDIA A100 Tensor Core GPU Architec- ture. https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/ nvidia-ampere-architecture-whitepaper.pdf . Accessed: 2025-03-10

    work page 2020

  7. [7]

    NVIDIA Corporation. 2023. NVIDIA DGX H100 System User Guide. https: //docs.nvidia.com/dgx/dgxh100-user-guide/dgxh100-user-guide.pdf . Ac- cessed: 2025-03-10

    work page 2023

  8. [8]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

Show all 48 references
  1. [9]

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al. 2021. DAPPLE: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of P...

    2021

  2. [10]

    Yangyang Feng, Minhui Xie, Zijie Tian, Shuo Wang, Youyou Lu, and Jiwu Shu. 2023. Mobius: Fine tuning large-scale models on commodity gpu servers. In Proceedings of the 28th ACM International Conference on Ar- chitectural Support for Programming Languages and Operating Systems,...

    2023

  3. [11]

    Swapnil Gandhi, Mark Zhao, Athinagoras Skiadopoulos, and Christos Kozyrakis. 2024. ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation. In Proceedings of the ACM SIGOPS 30th Symposium on Operat- ing Systems Principles. 211–228

    2024

  4. [12]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  5. [13]

    Muyan Hu, Ashwin Venkatram, Shreyashri Biswas, Balamurugan Marimuthu, Bohan Hou, Gabriele Oliaro, Haojie Wang, Liyan Zheng, Xupeng Miao, Jidong Zhai, and Zhihao Jia. 2024. Optimal Kernel Or- chestration for Tensor Programs with Korch. In Proceedings of the 29th ACM Internation...

    2024

  6. [14]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al

  7. [15]

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury

  8. [16]

    Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. 2019. TASO: optimizing deep learning computation with automatic generation of graph substitutions. InProceedings of the 27th ACM Symposium on Operating Systems Principles (Huntsville, Ontario...

    2019

  9. [17]

    Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. 2020. A unified architecture for accelerating distributed{DNN} train- ing in heterogeneous{GPU/CPU} clusters. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) . 463–479

    2020

  10. [18]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al

  11. [19]

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Re- ducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5 (2023), 341–353

    2023

  12. [20]

    Seonho Lee, Amar Phanishayee, and Divya Mahajan. 2025. Forecasting GPU Performance for Deep Learning Training and Inference. In Proceed- ings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam,...

    2025

  13. [21]

    Dacheng Li, Hongyi Wang, Eric Xing, and Hao Zhang. 2022. Amp: Auto- matically finding model parallel strategies with heterogeneity awareness. Advances in Neural Information Processing Systems 35 (2022), 6630–6639

    2022

  14. [22]

    In Proceedings of the 50th annual international symposium on computer architecture

    Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th annual international symposium on computer architecture . 1–14

  15. [23]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. 2020. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704 (2020)

    2020 arXiv

  16. [24]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  17. [25]

    Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. 2023. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model train- ing efficiency. In Proceedings of the International Conference for High Per- formance Computing, Networking, Storage and Analysis . 1–13

    2023

  18. [26]

    Shigang Li and Torsten Hoefler. 2021. Chimera: efficiently training large- scale neural networks with bidirectional pipelines. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–14

    2021

  19. [27]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large- scale language model training on gpu clusters using megatron-lm. In ...

    2021

  20. [28]

    Thomas Norrie, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi, and David Patterson. 2021. The Design Process for Google’s Training Chips: TPUv2 and TPUv3. IEEE Micro 41, 2 (2021), 56–63. doi:10.1109/MM.2021.3058217

    2021

  21. [29]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  22. [30]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. arXiv preprint arXiv:2211.13878 (2022)

    2022 arXiv

  23. [31]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Network- ing, Storage and Analysis. IEEE, 1–16

    2020

  24. [32]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining . ...

    2020

  25. [33]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  26. [34]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learn- ers. OpenAI blog 1, 8 (2019), 9

    2019

  27. [35]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeongjae Jeon

  28. [36]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  29. [37]

    Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, et al. [n. d.]. SimAI: Unify- ing Architecture Design and Performance Tunning for Large-Scale Large Language Model Training with Scalability and Precision. ([n. d.])....

    2025

  30. [38]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  31. [39]

    Mengdi Wu, Xinhao Cheng, Shengyu Liu, Chunan Shi, Jianan Ji, Kit Ao, Praveen Velliengiri, Xupeng Miao, Oded Padon, and Zhihao Jia

  32. [40]

    Yu, Yubo Gao, Pavel Golikov, and Gennady Pekhimenko

    Geoffrey X. Yu, Yubo Gao, Pavel Golikov, and Gennady Pekhimenko. 2021. Habitat: A Runtime-Based Computational Performance Predictor for Deep Neural Network Training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 503–521. https://www.usenix.org...

    2021

  33. [41]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 (2023)

    2023 arXiv

  34. [42]

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and{Intra-Operator} parallelism for distributed deep learning. In16th USENIX Symposium on Operating System...

    2022

  35. [43]

    Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: an insightful visual performance model for multicore architectures. Com- mun. ACM 52, 4 (April 2009), 65–76. doi: 10.1145/1498765.1498785

    2009

  36. [45]

    arXiv:2405.05751 [cs.LG] https://arxiv.org/abs/2405.05751

    Mirage: A Multi-Level Superoptimizer for Tensor Programs. arXiv:2405.05751 [cs.LG] https://arxiv.org/abs/2405.05751

    arXiv

  37. [2019]

    Advances in neural information processing systems 32 (2019)

    Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019)

    2019

  38. [2022]

    arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205.14135

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205.14135

    arXiv

  39. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles

    Oobleck: Resilient distributed training of large models using pipeline templates. In Proceedings of the 29th Symposium on Operating Systems Principles. 382–395

  40. [2024]

    In 2024 USENIX Annual Technical Conference (USENIX ATC 24)

    Metis: Fast Automatic Distributed Training on Heterogeneous {GPUs}. In 2024 USENIX Annual Technical Conference (USENIX ATC 24) . 563–578

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.