Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AxoNN sustains exascale bf16 throughput on three GPU supercomputers and shows that 70B-parameter models can memorize documents in a single pass.

desk verdict Systems half is strong and well-documented; the catastrophic-memorization threshold claim is confounded for the 405B and needs revision. read the letter →

arxiv 2502.08145 v1 pith:PLM7XXZ7 submitted 2025-02-12 cs.LG cs.AIcs.DC

Siddharth Singh , Prajwal Singhania , Aditya Ranjan , John Kirchenbauer , Jonas Geiping , Yuxin Wen , Neel Jain , Abhimanyu Hans
show 4 more authors
Manli Shu Aditya Tomar Tom Goldstein Abhinav Bhatele
This is my paper · ORCID
classification cs.LGcs.AIcs.DC
keywords four-dimensionalhybridparallelism3DparallelmatrixmultiplicationlargelanguagemodeltrainingexascalethroughputcatastrophicmemorizationGoldfishLossGPUsupercomputingweakscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents AxoNN, an open-source framework for training GPT-style transformers on GPU supercomputers, and reports sustained half-precision (bf16) throughput of 620.1 Petaflop/s on 4,096 A100 GPUs, 1.381 Exaflop/s on 32,768 MI250X GCDs, and 1.423 Exaflop/s on 6,144 H100 GPUs. The speed comes from a four-dimensional hybrid parallel scheme that combines data parallelism with a three-dimensional parallel matrix multiplication algorithm, plus automated BLAS kernel tuning, aggressive overlap of non-blocking collectives with computation, and a communication model that picks near-optimal GPU grid configurations. Using access to that scale, the paper also reports a memorization study: models at 70B parameters and above reproduce training documents verbatim after a single pass, with roughly 5% of documents memorized in one epoch, while 1B-13B models stay near zero. A masking technique called Goldfish Loss, which drops random tokens from the loss computation, brings exact-match rates back down to control levels. If these results hold, large-model training becomes accessible to academic and public researchers, and the same experiments expose a privacy and copyright risk that grows with model scale.

What carries the argument

The load-bearing object is the four-dimensional virtual GPU grid: GPUs are arranged as $G_{\text{data}} \times G_x \times G_y \times G_z$, where data parallelism runs across the $G_{\text{data}}$ groups and each group runs a three-dimensional parallel matrix multiplication scheme (derived from a classical 3D PMM algorithm) with the weight matrix sharded along the Z axis to save memory and all-gather, reduce-scatter, and all-reduce collectives moving data. A communication model assigns bandwidths to each hierarchy level and ranks configurations, while kernel tuning and overlap of non-blocking collectives with computation reduce time per batch. For memorization, the central mechanism is Goldfish Loss, which randomly masks tokens so that the cross-entropy loss is not computed on them, making exact reproduction at inference time very unlikely for the masked tokens.

What would settle it

Reproduce the 70B continued-training protocol with a held-out control bucket of 1,000 unseen documents and score memorization on 200-token spans; if the control bucket shows match rates as high as the trained buckets, or if longer-span matches vanish, the claimed single-pass catastrophic memorization would not be supported.

Watch

Extended reading notes

Core claim

The central discovery is that a four-dimensional hybrid parallel algorithm — data parallelism combined with a 3D parallel matrix multiplication over a $G_x \times G_y \times G_z$ grid — lets one open-source stack train GPT-style models with near-ideal weak scaling to thousands of GPUs and sustain high fractions of peak bf16 throughput on three systems: 620.1 Pflop/s on 4,096 A100 GPUs, 1.381 Eflop/s on 32,768 MI250X GCDs, and 1.423 Eflop/s on 6,144 H100 GPUs. The performance model predicts fast configurations by summing ring-collective times with bandwidths that depend on the process-group hierarchy, and the kernel-tuning and overlap optimizations close the remaining gap. The same scale enables a second finding: exact memorization of whole documents emerges sharply between the 13B and 70B parameter scales, with roughly 5% of documents memorized after one pass and up to 67% after six epochs in the 70B models, and Goldfish Loss masking suppresses this memorization to levels comparable to the control data.

Load-bearing premise

The memorization results rest on treating exact reproduction of the final 50 tokens of 200 Wikipedia articles as memorization caused by the continued-training runs, even though the 405B model already reproduced over 10% of the control documents before any continued training.

Editorial extensions

If this is right

  • Large-scale LLM pretraining and fine-tuning no longer requires proprietary stacks: a sequential training codebase can be parallelized with AxoNN and reach exascale throughput on government and academic supercomputers.
  • The paper's time-to-solution estimates project that an 80B model can be trained on 2 trillion tokens in about 25.5 days on 8,192 Frontier GCDs, and a 640B model in about 15 months.
  • Memorization risk is a scale effect, not a repetition effect alone: 70B models memorize documents in a single pass, so even perfectly deduplicated corpora can leak training data.
  • Goldfish Loss reduces exact-match rates of large models to control levels, providing a training-time mitigation for verbatim memorization.
  • AxoNN's four-dimensional algorithm generalizes existing parallel schemes: with certain grid axes it reduces to fully sharded data parallelism, ZeRO-style sharding, or Megatron-style tensor parallelism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's threshold claim (catastrophic memorization appears between 13B and 70B parameters) is based on a coarse size ladder; a sweep with intermediate sizes would locate the threshold and test whether it shifts with batch size, learning rate, or data domain.
  • Editorial extension: if single-pass memorization is real, deduplicated corpora remain vulnerable at scale, so mitigations like Goldfish masking may need to become a default part of training rather than an optional audit.
  • Editorial extension: the exascale figures were measured over short runs (ten iterations, averaged over the last eight); a longer sustained run on a dedicated partition would show whether the throughput holds over full training jobs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents AxoNN, an open-source framework for training large language models at scale, based on a four-dimensional hybrid parallel strategy that combines data parallelism with a three-dimensional parallel matrix-multiplication algorithm. It reports three main performance optimizations: automated BLAS kernel tuning, overlap of non-blocking collectives with computation, and a communication-cost model for selecting near-optimal process-grid configurations. The authors report sustained bf16 throughput of 620.1 Petaflop/s on 4,096 NVIDIA A100 GPUs, 1.381 Exaflop/s on 32,768 AMD MI250X GCDs, and 1.423 Exaflop/s on 6,144 NVIDIA H100 GPUs, with weak-scaling data across Perlmutter, Frontier, and Alps. The paper also presents a continued-pretraining study on Llama-family models claiming that 'catastrophic memorization' emerges at the 70B scale, with roughly 5% of documents memorized in one pass, and that the Goldfish Loss prevents this behavior. The performance model is validated against all possible configurations on 32 and 64 GPUs, and the FLOP count is checked against Nsight Compute.

Significance. If the performance results hold, AxoNN would be a notable open-source systems contribution: it demonstrates exascale bf16 throughput for LLM training on both NVIDIA and AMD GPU platforms, extends the previously demonstrated scale of distributed training, and provides a validated performance model plus a portable implementation. These strengths are substantial: the code is open source, the analytical FLOP count is checked empirically, and the performance model ranks nine of the ten fastest observed configurations correctly. The memorization study is scientifically interesting because it targets scales (70B and 405B) rarely accessible in academic settings, but its current experimental design does not yet support the size-threshold claim as cleanly as the systems contribution. The paper is best judged as two coupled contributions; the systems half is close to acceptance, while the memorization half needs additional analysis and controls.

major comments (3)
  1. [Section VIII-C, Figure 10] The claim that catastrophic memorization emerges at the 70B scale is not cleanly supported by the presented evidence. The 405B model already reproduces more than 10% of the control documents at the 0-epoch baseline, as the paper itself acknowledges, so the post-training rates for that model are partly pre-existing and the untreated control cannot serve as a clean counterfactual for the treated buckets. The paper does not report the 0-epoch baseline for the specific treated buckets, and the 70B 'one-pass 5%' figure is based on 200 documents and three trials, corresponding to roughly 10 documents. Please report the 0-epoch baseline for each treated bucket, add confidence intervals or per-document variability for the exact-match rates, and either restrict the threshold claim to a formal comparison between the 70B and smaller models or soften the conclusion accordingly.
  2. [Section VIII-D, Figure 11] The Goldfish-loss conclusion is weakened by the same pre-existing memorization issue. In Figure 11, the 405B Goldfish curve is compared against the 0-epoch control, but that control already contains memorized documents; reducing the post-training rate 'to control levels' therefore does not establish that new memorization was prevented, only that the rate after training is no higher than the pre-existing rate. To support the prevention claim, the authors should show the matched 0-epoch baseline for the same buckets that are later trained with Goldfish, and ideally report the per-document overlap between documents memorized before and after training.
  3. [Section VI-C and VII-B] The headline sustained flop/s numbers are based on a single run per configuration, averaged over the last 8 of 10 iterations, and Section VI-B acknowledges significant run-to-run variability due to network congestion and file-system degradation. For claims of 'sustained' exascale throughput and for the comparison in Table I, at least a small number of repeated runs (or, minimally, per-iteration min/max values) should be reported for the 4,096-GPU Perlmutter, 32,768-GCD Frontier, and 6,144-GPU Alps configurations. Without this, the 620.1 Petaflop/s, 1.381 Exaflop/s, and 1.423 Exaflop/s figures do not have a stated uncertainty.
minor comments (5)
  1. [Section VII-C] In the final paragraph of Section VII-C, 'rain these models' should read 'train these models'.
  2. [Section VI-C] The verification that Narayanan et al.'s analytical FLOP count matches Nsight Compute is described only as a 'small experiment'; please provide the experimental details or a reference so that the verification can be reproduced.
  3. [Section VI-C] The empirical peak of 813 Tflop/s for the GH200 is attributed to an NVIDIA benchmark guide without a citation; please add a reference or measurement details.
  4. [Section VIII-B] The decoding procedure for the memorization test is unspecified; please state whether generation is greedy or sampled, how many tokens are generated, and how truncation is handled, since the exact-match metric is sensitive to these choices.
  5. [References] References [13] and [27] are duplicates of the same ZeRO++ paper; remove one of them.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the performance model is validated against measured batch times, and the memorization/Goldfish results are controlled experiments rather than derivations from their own inputs.

full rationale

Walking the paper's main derivation chains, I find no step that reduces to its inputs by construction or by fitted-parameter renaming. The performance model (Section V-B) predicts communication times from analytical ring-collective formulas (Equations 1-7) plus independently profiled bandwidths, and is validated against the full set of measured batch times in Figure 2; the model is not fitted to those batch times, so the rank ordering is an independent check. The sustained FLOP/s figures (Sections VI-C and VII-B) use Narayanan et al.'s analytical FLOP count, which the authors verify against Nsight Compute on a small experiment, combined with timer measurements; no predicted performance number is obtained by inverting the reported result. The time-to-solution projections (Section VII-C) are simple extrapolations from measured per-iteration times at multiple GCD counts, not fitted predictions. The memorization study (Section VIII) uses separate treated and control buckets, with the control bucket explicitly reported, and the Goldfish-loss mitigation is applied from prior work [50] and evaluated with its own Exact Match measurements in Figure 11; the claim that masking reduces exact reproduction is a mechanistic consequence of the mask, but the paper presents it as an experimental demonstration rather than as a derived law, so it is not self-definitional in the prohibited sense. The self-citations to the authors' own AxoNN [9,10] and Goldfish [50] papers are routine prior-work citations and are not load-bearing: the current paper's measurements stand on their own. The paper itself notes the 405B anomaly is 'likely because we used one set of hyperparameters for all models' (Section VIII-C), and the reader-level concern about the 405B control bucket already showing over 10% exact match before continued training is a genuine interpretive confound for the size-threshold claim, but it is a correctness/experimental-design risk, not a circularity: it does not make any equation or conclusion equal to its own input by construction. Overall score 1 reflects no meaningful circularity in the derivation chain, with only routine self-citation present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central performance claims depend on a handful of empirical calibrations (bandwidth profiles, empirical peak flop/s) and standard assumptions about collective algorithms. The memorization claims depend on a specific metric and a single hyperparameter schedule; no new physical entities are introduced.

free parameters (5)
  • Intra-node bandwidth profile = profiled per process-group hierarchy with 1 GB messages
    Section V-B, Case 1: bandwidths for all intra-node 2D hierarchies are measured in advance and stored; the performance model retrieves these values to estimate collective times.
  • Inter-node bandwidth beta_inter = assumed uniform across node pairs
    Assumption-5 in Section V-B: a single value for all node-pair bandwidths, which is an approximation of the Slingshot topology.
  • Empirical peak flop/s = 280/125/813 Tflop/s for A100/MI250X/H100
    Section VI-C: used to report % of empirical peak; does not change absolute flop/s but affects efficiency interpretations.
  • Goldfish loss mask parameters = k=2, h=13
    Section VIII-D: chosen from the authors' prior work [50]; controls how many tokens are masked and the window.
  • Learning rate schedule = 3e-4 to 3e-5 decay
    Section VIII-B: single schedule used for all model sizes; the authors note extreme scales may need different hyperparameters.
assumptions (5)
  • domain assumption Ring algorithm for collectives
    Assumption-1 in Section V-B: all-reduce, reduce-scatter, and all-gather are modeled as ring-based, typical for NCCL/RCCL but not universally guaranteed.
  • domain assumption Message startup overhead is negligible
    Assumption-3 in Section V-B: transmission time is message size / bandwidth; valid for large messages but not for small ones.
  • standard math Analytical FLOP count formula matches actual operations
    Section VI-C: Narayanan et al.'s transformer FLOP formula is used for all flop/s numbers; a small Nsight Compute experiment validated it, but it remains an input assumption.
  • domain assumption Activation checkpointing does not change FLOP count
    Section VI-A: activation checkpointing is enabled, which recomputes activations and thus performs extra FLOPs not counted in the analytical formula; the paper ignores this overhead.
  • ad hoc to paper Exact 50-token continuation is a valid memorization metric
    Section VIII-B: memorization is defined as perfect reproduction of the last 50 tokens; this narrow metric is not compared with extraction-based memorization metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers." pith.science (2026). https://pith.science/paper/PLM7XXZ7

@misc{pith2026250208145,
  author       = {Pith},
  title        = {Pith review of: Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLM7XXZ7}},
  note         = {Machine review of arXiv:2502.08145}
}
read the original abstract

Training and fine-tuning large language models (LLMs) with hundreds of billions to trillions of parameters requires tens of thousands of GPUs, and a highly scalable software stack. In this work, we present a novel four-dimensional hybrid parallel algorithm implemented in a highly scalable, portable, open-source framework called AxoNN. We describe several performance optimizations in AxoNN to improve matrix multiply kernel performance, overlap non-blocking collectives with computation, and performance modeling to choose performance optimal configurations. These have resulted in unprecedented scaling and peak flop/s (bf16) for training of GPT-style transformer models on Perlmutter (620.1 Petaflop/s), Frontier (1.381 Exaflop/s) and Alps (1.423 Exaflop/s). While the abilities of LLMs improve with the number of trainable parameters, so do privacy and copyright risks caused by memorization of training data, which can cause disclosure of sensitive or private information at inference time. We highlight this side effect of scale through experiments that explore "catastrophic memorization", where models are sufficiently large to memorize training data in a single pass, and present an approach to prevent it. As part of this study, we demonstrate fine-tuning of a 405-billion parameter LLM using AxoNN on Frontier.

Figures

Figures reproduced from arXiv: 2502.08145 by the authors.

Figure 1
Figure 1. Parallelization of a matrix multiply in an FC layer with Agarwal’s [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Plots validating the performance model by comparing the observed time per batch and the rank ordered by the model for two neural networks: [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Creation of a ring among eight GPUs on two nodes for a collective [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Two rings among four GPUs each across two nodes for performing [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The impact of overlapping non-blocking collectives with computation [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Weak scaling performance (time per batch or iteration) of AxoNN on [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Sustained flop/s on different platforms. The FLOP count is calculated [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Strong scaling showing expected time-to-solution on Frontier. Using [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Memorization as a function of parameter count and epochs (repetitions of the training data). For each model size, we show the “Exact Match” rate [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: The impact of applying Goldfish Loss during training to mitigate [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 40 canonical work pages

  1. [1]

    Super: Sub-graph parallelism for transformers,

    A. Jain, T. Moon, T. Benson, H. Subramoni, S. A. Jacobs, D. K. Panda, and B. V . Essen, “Super: Sub-graph parallelism for transformers,” in 2021 IEEE International Parallel and Distributed Processing Sympo- sium (IPDPS), 2021, pp. 629–638

    work page 2021

  2. [2]

    Scaling distributed deep learning work- loads beyond the memory capacity with karma,

    M. Wahib, H. Zhang, T. T. Nguyen, A. Drozd, J. Domke, L. Zhang, R. Takano, and S. Matsuoka, “Scaling distributed deep learning work- loads beyond the memory capacity with karma,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’20. IEEE Press, 2020

    work page 2020

  3. [3]

    Forge: Pre-training open foundation models for science,

    J. Yin, S. Dash, F. Wang, and M. Shankar, “Forge: Pre-training open foundation models for science,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’23. New York, NY , USA: Association for Computing Machinery, 2023

    work page 2023

  4. [4]

    Optimizing distributed training on frontier for large language models,

    S. Dash, I. R. Lyngaas, J. Yin, X. Wang, R. Egele, J. A. Ellis, M. Maiterth, G. Cong, F. Wang, and P. Balaprakash, “Optimizing distributed training on frontier for large language models,” in ISC High Performance 2024 Research Paper Proceedings (39th International Conference), 2024, pp. 1–11

    work page 2024

  5. [5]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,

    S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V . Korthikanti, E. Zhang, R. Child, R. Y . Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y . He, M. Houston, S. Tiwary, and B. Catanzaro, “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” Tec...

    work page 2022

  6. [6]

    Efficient large-scale language model training on GPU clusters,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters,” CoRR, vol. abs/2104.04473, 2021

    arXiv 2021

  7. [7]

    MegaScale: Scaling large language model training to more than 10,000 GPUs,

    Z. Jiang, H. Lin, Y . Zhong, Q. Huang, Y . Chen, Z. Zhang, Y . Peng, X. Li, C. Xie, S. Nong, Y . Jia, S. He, H. Chen, Z. Bai, Q. Hou, S. Yan, D. Zhou, Y . Sheng, Z. Jiang, H. Xu, H. Wei, Z. Zhang, P. Nie, L. Zou, S. Zhao, L. Xiang, Z. Liu, Z. Li, X. Jia, J. Ye, X. Jin, and X. Liu, “MegaScale: Scaling large language model training to more than 10,000 GPUs,...

    work page 2024

  8. [8]

    Google cloud demonstrates the world’s largest distributed training job for large language models across 50000+ tpu v5e chips,

    Google, “Google cloud demonstrates the world’s largest distributed training job for large language models across 50000+ tpu v5e chips,” https://cloud.google.com/blog/products/compute/ the-worlds-largest-distributed-llm-training-job-on-tpu-v5e

    work page

Show all 49 references
  1. [9]

    AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning,

    S. Singh and A. Bhatele, “AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning,” in Proceedings of the IEEE International Parallel & Distributed Processing Symposium , ser. IPDPS ’22. IEEE Computer Society, May 2022

    2022

  2. [10]

    Exploiting sparsity in pruned neural networks to optimize large model training,

    ——, “Exploiting sparsity in pruned neural networks to optimize large model training,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 245–255. [Online]. Available: https://doi.ieeeco...

    2023

  3. [11]

    Zero: Memory optimizations toward training trillion parameter models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’20. IEEE Press, 2020

    2020

  4. [12]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel,

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Dama- nia, B. Nguyen, G. Chauhan, Y . Hao, A. Mathews, and S. Li, “Pytorch fsdp: Experiences on scaling fully sharded data parallel,” Proc. VLDB...

    2023

  5. [14]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” Tech. Rep., 2020

    2020

  6. [15]

    GPipe: efficient training of giant neural networks using pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and z. Chen, “GPipe: efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019

    2019

  7. [16]

    Deepspeed: Extreme-scale model training for everyone,

    Microsoft, “Deepspeed: Extreme-scale model training for everyone,” https://www.microsoft.com/en-us/research/blog/ deepspeed-extreme-scale-model-training-for-everyone/

  8. [17]

    A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training,

    S. Singh, O. Ruwase, A. A. Awan, S. Rajbhandari, Y . He, and A. Bhatele, “A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training,” in Proceedings of the 37th International Conference on Supercomputing, ser. ICS ’23. New York, NY , USA: Associa...

    2023

  9. [18]

    GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch,

    A. Andonian et al. , “GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch,” 9 2023. [Online]. Available: https://www.github.com/eleutherai/gpt-neox

    2023

  10. [19]

    Alpa: Automating inter- and intra-operator parallelism for distributed deep learning,

    L. Zheng, Z. Li, H. Zhang, Y . Zhuang, Z. Chen, Y . Huang, Y . Wang, Y . Xu, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Alpa: Automating inter- and intra-operator parallelism for distributed deep learning,” CoRR, vol. abs/2201.12023, 2022

    2022 arXiv

  11. [20]

    Colossal-AI: a unified deep learning system for large-scale parallel training,

    S. Li, H. Liu, Z. Bian, J. Fang, H. Huang, Y . Liu, B. Wang, and Y . You, “Colossal-AI: a unified deep learning system for large-scale parallel training,” in Proceedings of the 52nd International Conference on Parallel Processing, ser. ICPP ’23. New York, NY , USA: Association...

    2023

  12. [21]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron et al. , “Llama 2: Open foundation and fine-tuned chat models,” Tech. Rep., 2023

    2023

  13. [22]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2019. [Online]. Available: https://arxiv.org/abs/1910.10683

    2019 arXiv

  14. [23]

    LBANN: livermore big artificial neural network HPC toolkit,

    B. V . Essen, H. Kim, R. A. Pearce, K. Boakye, and B. Chen, “LBANN: livermore big artificial neural network HPC toolkit,” in Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, MLHPC 2015, Austin, Texas, USA, November 15, 2015 . ACM, 201...

    2015

  15. [24]

    Nvidia selene supercomputer,

    “Nvidia selene supercomputer,” https://www.top500.org/system/ 179842/

  16. [25]

    Frontier: Exploring exascale,

    S. Atchley et al., “Frontier: Exploring exascale,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’23. New York, NY , USA: Association for Computing Machinery, 2023

    2023

  17. [26]

    A three-dimensional approach to parallel matrix multiplication,

    R. C. Agarwal, S. M. Balle, F. G. Gustavson, M. Joshi, and P. Palkar, “A three-dimensional approach to parallel matrix multiplication,” IBM Journal of Research and Development, vol. 39, no. 5, pp. 575–582, 1995

    1995

  18. [27]

    ZeRO++: Extremely efficient collective communication for large model training,

    G. Wang, H. Qin, S. A. Jacobs, X. Wu, C. Holmes, Z. Yao, S. Rajbhandari, O. Ruwase, F. Yan, L. Yang, and Y . He, “ZeRO++: Extremely efficient collective communication for large model training,” in The Twelfth International Conference on Learning Representations, 2024. [Online]...

    2024

  19. [28]

    Improving the performance of collective operations in mpich,

    R. Thakur and W. D. Gropp, “Improving the performance of collective operations in mpich,” in Recent Advances in Parallel Virtual Machine and Message Passing Interface , J. Dongarra, D. Laforenza, and S. Or- lando, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 257–267

    2003

  20. [29]

    Optimization of collective reduction operations,

    R. Rabenseifner, “Optimization of collective reduction operations,” in Computational Science - ICCS 2004 , M. Bubak, G. D. van Albada, P. M. A. Sloot, and J. Dongarra, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp. 1–9

    2004

  21. [30]

    Improving communication performance in dense linear algebra via topology aware collectives,

    E. Solomonik, A. Bhatele, and J. Demmel, “Improving communication performance in dense linear algebra via topology aware collectives,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’11. ACM, Nov. 2011,...

    2011

  22. [31]

    Mapping applications with collectives over sub-communicators on torus networks,

    A. Bhatele, T. Gamblin, S. H. Langer, P.-T. Bremer, E. W. Draeger, B. Hamann, K. E. Isaacs, A. G. Landge, J. A. Levine, V . Pascucci, M. Schulz, and C. H. Still, “Mapping applications with collectives over sub-communicators on torus networks,” in Proceedings of the ACM/IEEE In...

    2012 doi

  23. [32]

    RAHTM: Routing- algorithm aware hierarchical task mapping,

    A. Abdel-Gawad, M. Thottethodi, and A. Bhatele, “RAHTM: Routing- algorithm aware hierarchical task mapping,” in Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’14. IEEE Computer Society, Nov. 2014...

    2014 doi

  24. [33]

    Optimizing the performance of parallel applications on a 5D torus via task mapping,

    A. Bhatele, N. Jain, K. E. Isaacs, R. Buch, T. Gamblin, S. H. Langer, and L. V . Kale, “Optimizing the performance of parallel applications on a 5D torus via task mapping,” in Proceedings of IEEE International Conference on High Performance Computing , ser. HiPC ’14. IEEE Comp...

    2014

  25. [34]

    Supervised learning based algorithm selection for deep neural networks,

    S. Shi, P. Xu, and X. Chu, “Supervised learning based algorithm selection for deep neural networks,” in 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS) , 2017, pp. 344–351

    2017

  26. [35]

    Language models are few-shot learners,

    T. B. Brown et al. , “Language models are few-shot learners,” CoRR, vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/ 2005.14165

    2005 arXiv

  27. [36]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” CoRR, vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762

    2017 arXiv

  28. [37]

    Bigscience large open-science open-access multilingual language model,

    BigScience, “Bigscience large open-science open-access multilingual language model,” https://huggingface.co/bigscience/bloom, 2022

    2022

  29. [38]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” Tech. Rep., 2019

    2019

  30. [39]

    Training deep nets with sublinear memory cost,

    T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” CoRR, vol. abs/1604.06174, 2016. [Online]. Available: http://arxiv.org/abs/1604.06174

    2016 arXiv

  31. [40]

    A study of BFLOAT16 for deep learning training,

    D. D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. V ooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey, “A study of BFLOAT16 for deep learning...

    1905 arXiv

  32. [41]

    Lightning AI, “Litgpt,” https://github.com/Lightning-AI/litgpt, 2023

    2023

  33. [42]

    Interactive investigation of traffic congestion on fat-tree networks using TreeScope,

    H. Bhatia, N. Jain, A. Bhatele, Y . Livnat, J. Domke, V . Pascucci, and P.-T. Bremer, “Interactive investigation of traffic congestion on fat-tree networks using TreeScope,” Computer Graphics Forum, vol. 37, no. 3, pp. 561–572, Jun. 2018

    2018

  34. [43]

    Quantifying I/O and communication traffic interference on dragonfly networks equipped with burst buffers,

    M. Mubarak, P. Carns, J. Jenkins, J. Li, N. Jain, S. Snyder, R. B. Ross, C. D. Carothers, A. Bhatele, and K.-L. Ma, “Quantifying I/O and communication traffic interference on dragonfly networks equipped with burst buffers,” in Proceedings of the IEEE Cluster Conference , ser. ...

    2017

  35. [44]

    Quantifying memorization across neural language models,

    N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=TatRHT 1cK

    2023

  36. [45]

    The times sues openai and microsoft over ai use of copyrighted work,

    M. M. Grynbaum and R. Mac, “The times sues openai and microsoft over ai use of copyrighted work,” The New York Times , vol. 27, 2023

    2023

  37. [46]

    Extracting training data from large language models,

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in 30th USENIX Security Symposium (USENIX Security 21) , 2021, pp. 2633–2650

    2021

  38. [47]

    Pythia: A suite for analyzing large language models across training and scaling,

    S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff et al. , “Pythia: A suite for analyzing large language models across training and scaling,” in International Conference on Machine Learning . PMLR, ...

    2023

  39. [48]

    Tinyllama: An open-source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” Tech. Rep., 2024

    2024

  40. [49]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” Tech. Rep., 2024

    2024

  41. [50]

    Be like a goldfish, don’t memorize! mitigating memorization in generative llms,

    A. Hans, Y . Wen, N. Jain, J. Kirchenbauer, H. Kazemi, P. Singhania, S. Singh, G. Somepalli, J. Geiping, A. Bhatele et al., “Be like a goldfish, don’t memorize! mitigating memorization in generative llms,” arXiv preprint arXiv:2406.10209, 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.