Pith. sign in

REVIEW 2 major objections 5 minor 34 references

Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-level sharding scheme confines weight and gradient traffic to the fastest links on Frontier, yielding a 1.71x throughput gain over ZeRO++ at 384 GCDs.

desk verdict A competent Frontier-specific engineering extension to ZeRO++ with plausible but under-evidenced speedups; the communication-volume accounting is incomplete and needs a second pass. read the letter →

arxiv 2501.04266 v2 pith:V5TJCKKI submitted 2025-01-08 cs.DC cs.AI

Lang Xu , Quentin Anthony , Jacob Hatef , Aamir Shafi , Hari Subramoni , Dhabaleswar K. (DK) Panda This is my paper · ORCID
classification cs.DCcs.AI
keywords LLMtrainingZeRO++hierarchicalpartitioningcommunicationoptimizationAMDMI250XFrontiersupercomputergradientquantizationlow-bandwidth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model training with ZeRO-3 shards weights, gradients, and optimizer states across all workers, but every training step then pays for expensive node-crossing Allgather and Reduce-scatter operations. This paper claims that on Frontier, those collectives can be deliberately confined to the fast layers: model weights live on the two compute dies inside one MI250X GPU, gradients live on the eight dies inside one node, and only optimizer states are sharded across all nodes. The measured result is a 1.71x increase in TFLOPS per GPU over ZeRO++ for a 20B-parameter GPT model at 384 GCDs, with scaling efficiency of 0.94. If the claim holds, it shows that topology-aware sharding plus block quantization can turn a low-bandwidth supercomputer interconnect into a near-linear scaling platform for billion-parameter training.

What carries the argument

The central object is a three-level hierarchical partition indexed by Frontier's three bandwidth layers: GCD-to-GCD inside an MI250X (fastest), GPU-to-GPU inside a node via Infinity Fabric, and inter-node Slingshot (slowest). The scheme fixes the weight shard factor at 2, the gradient shard factor at 8 (one node), and the optimizer-state shard factor at the total number of GCDs. These choices satisfy the dependency rule that each worker only holds gradients and optimizer states for parameters it owns, and they make the device count for weight and gradient collectives independent of job size. Communication volume is further cut with block-wise quantization: INT8 for weight Allgather and secondary partitions, INT4 for gradient Reduce-scatter.

What would settle it

Instrument the cross-node gradient Allreduce separately during a 384-GCD 20B run and measure its time as a function of node count; if its cost scales with the number of nodes rather than staying near zero or hidden behind compute, the volume table understates communication and the scaling claim fails.

Watch

Extended reading notes

Core claim

The central claim is that ZeRO-3's communication bottleneck on Frontier is avoidable by assigning each model state a sharding factor matched to a bandwidth tier. The proposed scheme shards FP16 model weights across the two GCDs inside a single MI250X package (shard degree 2), shards gradients across the eight GCDs inside one compute node (shard degree 8), and shards optimizer states across all GCDs in the job, mirroring ZeRO-3. Weight Allgather for forward and backward passes then runs only over the fast GCD-to-GCD links inside one MI250X, gradient Reduce-scatter runs only over intra-node links, and INT8/INT4 block quantization halves or quarters the bytes moved. The paper reports up to 139.8% higher TFLOPS per GPU than ZeRO-3 and 70.7% higher than ZeRO++ for a 20B model at 384 GCDs, a 1.71x gain over ZeRO++, and scaling efficiency of 0.94, with loss curves close to unquantized ZeRO-3.

Load-bearing premise

The analysis counts only the intra-node Reduce-scatter volume for gradients and silently assumes the cross-node Allreduce that synchronizes gradient shards across nodes is cheap or fully overlapped; if that Allreduce grows with node count, the claimed communication savings and the 0.94 scaling efficiency do not follow.

Editorial extensions

If this is right

  • At 384 GCDs a 20B-parameter model trains at 1.71x the TFLOPS per GPU of ZeRO++, and scaling efficiency reaches 0.94, so jobs that previously slowed at large node counts can run near-linearly.
  • Weight Allgather latency becomes constant as nodes are added because it touches only two GCDs, and gradient Reduce-scatter latency becomes constant because it stays inside a node.
  • Block quantization keeps the training loss close to unquantized ZeRO-3 while reducing bytes moved, so the throughput gain does not come at an obvious convergence cost in the measured runs.
  • The scheme trades memory for bandwidth: weights and gradients occupy fixed per-device memory (1.5ψ and ψ/4 bytes), which lets large models run on small node counts but caps practical model size near 36B parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only validates on Frontier with AMD MI250X; the same shard-degree recipe would need recalibration on systems with different bandwidth ratios, and the general lesson is the ratio, not the absolute numbers.
  • The volume analysis omits the cross-node Allreduce that synchronizes gradient shards across nodes; whether that call stays hidden behind computation is the main untested scaling risk beyond 384 GCDs.
  • Because per-device weight and gradient memory is fixed, comparing this scheme with ZeRO-3 involves a crossover: at small node counts ZeRO-3 uses less per-device memory, while at large node counts this scheme avoids ZeRO-3's growing inter-node traffic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a topology-aware extension of ZeRO++ for LLM training on the Frontier supercomputer, based on a three-level hierarchical partitioning of weights (across two GCDs), gradients (across eight GCDs within a node), and optimizer states (across all GCDs), combined with block-based quantization for communication and secondary weight partitions. The authors report a 1.71x increase in TFLOPS per GPU over ZeRO++ and a scaling efficiency of 0.94 for a 20B GPT model at up to 384 GCDs, as well as convergence loss curves matching ZeRO-3. The paper also describes a porting of ZeRO++ to AMD GPUs and a hardware topology analysis of the DGX-A100 versus Frontier nodes.

Significance. If the measured performance holds, the paper is a useful systems contribution: it appears to be the first adaptation of ZeRO++ to AMD GPUs, it exploits Frontier's three-level bandwidth hierarchy, and it applies quantization to reduce-scatter as well as allgather collectives. The inclusion of convergence curves and scaling studies strengthens the empirical case. However, the central communication-cost argument is incomplete, because the cross-node Allreduce required by the design is absent from the volume analysis; this gap directly affects the scaling-efficiency claim. The paper would be suitable for a systems venue after the missing communication term is accounted for and the evaluation methodology is clarified.

major comments (2)
  1. [V.D, Tables VII-VIII; V.C] The communication-volume analysis in §V.D omits the cross-node Allreduce that the design requires. Section V.C states that after the intra-node Reduce-scatter, "we call Allreduce on local gradients stored among nodes" to synchronize gradient shards before weight updates, because optimizer states are sharded across all GCDs while weights are replicated only across pairs of GCDs. This Allreduce moves inter-node traffic that scales with node count and with the gradient shard size; it is not included in Table VIII, nor is any overlap or latency-hiding argument provided. The paper must either add this term to the volume model or show empirically that it is fully hidden; otherwise the claimed inter-node communication reduction and the 0.94 scaling efficiency in Figure 7b do not follow from the stated analysis.
  2. [VI, Figure 7] The scaling-efficiency metric is never defined. The abstract and Figure 7b report a scaling efficiency of 0.94 at up to 384 GCDs, but no baseline configuration is specified (e.g., efficiency relative to 8, 24, or 48 GCDs), and no raw TFLOPS/GPU numbers or repeated-run variance are given for any configuration. Please state the formula and baseline, and report per-configuration measurements with at least the run-to-run spread, since the central speedup and efficiency claims rest on these points.
minor comments (5)
  1. [V.A, Table V] The text says each GCD hosts 1.5ψ bytes of weight memory, but with Sec-Degree=8 the table gives 2ψ/2 + ψ/8 = 1.125ψ bytes; 1.5ψ corresponds to Sec-Degree=2. Please reconcile the text with the table.
  2. [Tables V and VIII] There are typographical artifacts "ψψψ" in several formulas (e.g., "2ψψψ/8" and "ψψψ/2") that should read "2ψ/8" and "ψ/2".
  3. [References] References [13] and [27] are the same paper (PyTorch FSDP) and should be merged or differentiated.
  4. [VII.C] The paper lists MFU as future work, but adding MFU would help substantiate the efficiency claims beyond TFLOPS per GPU.
  5. [Figures 7 and 8] The y-axes and legends lack units; please specify whether TFLOPS per GPU is measured, theoretical-peak-normalized, and how samples per second is defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical measurements against external baselines and its communication-volume analysis is not fitted to those results.

full rationale

The paper's central claims are empirical speedups and scaling-efficiency measurements for a proposed 3-level hierarchical partitioning strategy on Frontier, benchmarked against ZeRO-3 and ZeRO++ as external baselines. The communication-volume analysis in Tables VII and VIII is a design-stage accounting of per-collective traffic based on chosen sharding factors (weights across 2 GCDs, gradients across 8 GCDs per node, optimizer states across all GCDs); these sharding factors are justified by the Frontier topology measurements in Section IV, not fitted to the reported TFLOPS numbers. The reported 1.71x speedup over ZeRO++ and 0.94 scaling efficiency are measured outcomes, not quantities derived from the paper's own formulas in a way that would make them true by construction. Convergence is validated by comparing loss curves against standard ZeRO-3 training, which is an external reference. The paper does rely on ZeRO++'s quantization-convergence result, but that is prior external work with its own independent evaluation, and the paper additionally presents its own loss-curve comparisons. No parameter is fitted to a subset of data and then presented as a prediction. One reviewer-identified correctness concern, namely that the cross-node gradient Allreduce mentioned in Section V.C is not included in the gradient communication-volume table, is an omission or modeling gap, not a circularity: it does not make the claimed speedup an input to the derivation. Therefore the derivation chain is self-contained with respect to the claims it makes, and no circular step can be exhibited from the paper's text.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the specified Frontier topology bandwidths, standard collective-communication cost formulas, and an unquantified assumption about the cost of the cross-node gradient Allreduce. The only hand-selected values are the shard degrees (2, 8, 8 for secondary weights), which are design choices rather than fitted constants. No new theoretical entities are introduced.

free parameters (3)
  • Weight shard degree (P_w = 2 GCDs) = 2
    Chosen by hand from Frontier topology analysis to keep weight Allgather within an MI250X; not fitted to measured data.
  • Gradient shard degree (P_g = 8 GCDs) = 8
    Chosen to confine gradient Reduce-scatter to intra-node communication, based on the bandwidth hierarchy; not fitted to data.
  • Secondary weight partition degree (Sec-Degree = 8) = 8
    Selected by hand for the quantized secondary weight partitions; the paper also considers Sec-Degree=2 but does not report experiments with it.
assumptions (4)
  • domain assumption Frontier hardware bandwidths match the paper's description: 200 GB/s GCD-GCD, 50-100 GB/s GPU-GPU, 100 GB/s inter-node.
    Invoked in Section IV to justify choosing P_w=2 and P_g=8; if the real achievable RCCL bandwidths differ, the optimal partition may be different.
  • standard math Collective communication time is dominated by the volume formulas in Tables VII and VIII (for example Allgather volume approximately psi*(d-1)/d on d devices).
    Used in Section V.D to argue communication reduction; this model omits the cross-node gradient Allreduce and does not model latency or overlap.
  • domain assumption Block-based quantization to INT8 weights and INT4 gradients preserves convergence close to FP16 training.
    Borrowed from ZeRO++ [18] and tested here for two GPT-NeoX runs up to 14B tokens with loss curves; no final-loss table is provided.
  • domain assumption AMSP's training-parameter dependency rule (optimizer-state shard factor >= gradient shard factor >= weight shard factor) is a correct constraint.
    Stated in Section V to justify sharding gradients at degree 8 while weights are at degree 2; no proof or sensitivity analysis is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning." pith.science (2026). https://pith.science/paper/V5TJCKKI

@misc{pith2026250104266,
  author       = {Pith},
  title        = {Pith review of: Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5TJCKKI}},
  note         = {Machine review of arXiv:2501.04266}
}
read the original abstract

Scaling up Large Language Model(LLM) training involves fitting a tremendous amount of training parameters across a limited number of workers. However, methods like ZeRO-3 that drastically reduce GPU memory pressure often incur heavy communication to ensure global synchronization and consistency. Established efforts such as ZeRO++ use secondary partitions to avoid inter-node communications, given that intra-node GPU-GPU transfer generally has more bandwidth and lower latency than inter-node connections. However, as more capable infrastructure like Frontier, equipped with AMD GPUs, emerged with impressive computing capability, there is a need for investigations on the hardware topology and to develop targeted strategies to improve training efficiency. In this work, we propose a collection of communication and optimization strategies for ZeRO++ to reduce communication costs and improve memory utilization. In this paper, we propose a 3-level hierarchical partitioning specifically for the current 2nd ranked supercomputing cluster, Frontier, which aims at leveraging various bandwidths across layers of communications (GCD-GCD, GPU-GPU, and inter-node) to reduce communication overhead. For a 20B GPT model, we observe a 1.71x increase in TFLOPS per GPU when compared with ZeRO++ up to 384 GCDs and a scaling efficiency of 0.94 for up to 384 GCDs.

Figures

Figures reproduced from arXiv: 2501.04266 by the authors.

Figure 1
Figure 1. ZeRO-3 across two Frontier nodes. ZeRO++ [18] employs quantization kernels to reduce mes￾sage size and secondary partitioning to eliminate inter-node Allgather. To verify the effectiveness of ZeRO++ on Frontier, we ported it onto AMD GPUs and collected system throughput and max model size reachable. We observed that because ZeRO++ retains secondary parameter partitions within a node, the model size permitted can dec… view at source ↗
Figure 2
Figure 2. Topology of a DGX A100 compute node [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Topology of a compute node on ORNL Frontier [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Gradient partition communication in each step. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Proposed 3-level partitioning with one Frontier compute node. In this diagram, each GCD will hold half of the FP16 model weights and quantized [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparing TFLOPS per GPU across scales and scaling efficiency with GPT-NeoX-20B ZeRO-topo towards naive ZeRO-3 and ZeRO++ [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparing TFLOPS per GPU across scales and scaling efficiency with GPT-NeoX-10B ZeRO-topo towards naive ZeRO-3 and ZeRO++ [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Loss curve for ZeRO-Topo vs ZeRO-3 with GPT-NeoX-20B [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 9
Figure 9. Figure 9: Loss curve for ZeRO-Topo vs ZeRO-3 with GPT-NeoX-10B [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 14 canonical work pages

  1. [1]

    Introducing the next generation of Claude — anthropic.com,

    “Introducing the next generation of Claude — anthropic.com,” https: //www.anthropic.com/news/claude-3-family, [Accessed 06-06-2024]

    work page 2024

  2. [2]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivià ´lre, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. HÃl’liou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ipp...

    arXiv 2024

  3. [3]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  4. [4]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    arXiv 2021

  5. [5]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 2021. [Online]. Available: https://arxiv.org/abs/2110.14168

    arXiv 2021

  6. [6]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,”

  7. [7]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” 2021. [Online]. Available: https://arxiv.org/abs/2104.04473 10 Related Works Hybrid Sharding Frontier-...

    arXiv 2021

  8. [8]

    The mvapich project: Transforming research into high-performance mpi library for hpc community,

    D. K. Panda, H. Subramoni, C.-H. Chu, and M. Bayatpour, “The mvapich project: Transforming research into high-performance mpi library for hpc community,” Journal of Computational Science , vol. 52, p. 101208, 2021, case Studies in Translational Computer Science. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S1877750320305093

    work page 2021

Show all 34 references
  1. [9]

    Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation,

    A. A. A. et al, “Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation,” in 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) , 2019, pp. 498–507

    2019

  2. [10]

    NVIDIA Collective Communications Library (NCCL),

    NVIDIA, “NVIDIA Collective Communications Library (NCCL),” https://developer.nvidia.com/nccl, 2024, accessed: February 5, 2025

    2024

  3. [11]

    Amsp: Reducing communication overhead of zero for efficient llm training,

    Q. Chen, Q. Hu, G. Wang, Y . Xiong, T. Huang, X. Chen, Y . Gao, H. Yan, Y . Wen, T. Zhang, and P. Sun, “Amsp: Reducing communication overhead of zero for efficient llm training,” 2024. [Online]. Available: https://arxiv.org/abs/2311.00257

    2024 arXiv

  4. [12]

    Zero: Memory optimizations toward training trillion parameter models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” 2020. [Online]. Available: https://arxiv.org/abs/1910.02054

    2020 arXiv

  5. [14]

    Fairscale: A general purpose modular pytorch li- brary for high performance and large scale training,

    FairScale authors, “Fairscale: A general purpose modular pytorch li- brary for high performance and large scale training,” https://github.com/ facebookresearch/fairscale, 2021

    2021

  6. [15]

    Megatron-LM: Ongoing research training transformer models at scale,

    NVIDIA, “Megatron-LM: Ongoing research training transformer models at scale,” https://github.com/NVIDIA/Megatron-LM, 2024, accessed: February 5, 2025

    2024

  7. [16]

    Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 | TOP500 — top500.org,

    “Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 | TOP500 — top500.org,” https://www.top500.org/system/180047/, [Accessed 10-06- 2024]

    2024

  8. [17]

    An in-depth analysis of the slingshot interconnect,

    D. De Sensi, S. Di Girolamo, K. H. McMahon, D. Roweth, and T. Hoefler, “An in-depth analysis of the slingshot interconnect,” inSC20: International Conference for High Performance Computing, Networking, Storage and Analysis , 2020, pp. 1–14

    2020

  9. [18]

    Zero++: Extremely efficient collective communication for giant model training,

    G. Wang, H. Qin, S. A. Jacobs, C. Holmes, S. Rajbhandari, O. Ruwase, F. Yan, L. Yang, and Y . He, “Zero++: Extremely efficient collective communication for giant model training,” 2023. [Online]. Available: https://arxiv.org/abs/2306.10209

    2023 arXiv

  10. [19]

    Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,

    T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,” 2018. [Online]. Available: https://arxiv.org/abs/1802.09941

    2018 arXiv

  11. [20]

    Scaling single- image super-resolution training on modern hpc clusters: Early experi- ences,

    Q. Anthony, L. Xu, H. Subramoni, and D. K. D. Panda, “Scaling single- image super-resolution training on modern hpc clusters: Early experi- ences,” in 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2021, pp. 923–932

    2021

  12. [21]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017

    2017

  13. [22]

    8-bit optimizers via block-wise quantization,

    T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” 2022. [Online]. Available: https://arxiv.org/abs/2110.02861

    2022 arXiv

  14. [23]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 2019

    2019

  15. [24]

    GPT-NeoX-20B: An open-source autoregressive language model,

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang et al. , “GPT-NeoX-20B: An open-source autoregressive language model,” in Proceedings of BigScience Episode #5–Workshop on Challenges & Perspectives in Creating Large Lan...

    2022

  16. [25]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. C...

    2020

  17. [26]

    Rwkv: Reinventing rnns for the transformer era,

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV , X. He, H. Hou, J. Lin, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, B. Wang, J. S. Wind, S. Woz...

    2023 arXiv

  18. [27]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel,

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y . Hao, A. Mathews, and S. Li, “Pytorch fsdp: Experiences on scaling fully sharded data parallel,”

  19. [28]

    Mics: Near-linear scaling for training gigantic model on public cloud,

    Z. Zhang, S. Zheng, Y . Wang, J. Chiu, G. Karypis, T. Chilimbi, M. Li, and X. Jin, “Mics: Near-linear scaling for training gigantic model on public cloud,” 2022. [Online]. Available: https://arxiv.org/abs/2205. 00119

    2022

  20. [29]

    Megascale: Scaling large language model training to more than 10,000 gpus,

    Z. Jiang, H. Lin, Y . Zhong, Q. Huang, Y . Chen, Z. Zhang, Y . Peng, X. Li, C. Xie, S. Nong, Y . Jia, S. He, H. Chen, Z. Bai, Q. Hou, S. Yan, D. Zhou, Y . Sheng, Z. Jiang, H. Xu, H. Wei, Z. Zhang, P. Nie, L. Zou, S. Zhao, L. Xiang, Z. Liu, Z. Li, X. Jia, J. Ye, X. Jin, and X. ...

    2024 arXiv

  21. [30]

    Available: https://arxiv.org/abs/2304.11277

    [Online]. Available: https://arxiv.org/abs/2304.11277

    arXiv

  22. [31]

    Optimizing distributed training on frontier for large language models,

    S. Dash, I. Lyngaas, J. Yin, X. Wang, R. Egele, G. Cong, F. Wang, and P. Balaprakash, “Optimizing distributed training on frontier for large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2312.12705

    2023 arXiv

  23. [32]

    Comparative study of large language model architectures on frontier,

    J. Yin, A. Bose, G. Cong, I. Lyngaas, and Q. Anthony, “Comparative study of large language model architectures on frontier,” 2024. [Online]. Available: https://arxiv.org/abs/2402.00691

    2024 arXiv

  24. [33]

    Accelerating large language model training with hybrid gpu-based compression,

    L. Xu, Q. Anthony, Q. Zhou, N. Alnaasan, R. Gulhane, A. Shafi, H. Sub- ramoni, and D. Panda, “Accelerating large language model training with hybrid gpu-based compression,” in IEEEACM International Symposium on Cluster, Cloud, and Internet Computing 2024 , May 2024

    2024

  25. [36]

    Democratizing ai: Open-source scalable llm training on gpu-based supercomputers,

    S. Singh, P. Singhania, A. Ranjan, J. Kirchenbauer, J. Geiping, Y . Wen, N. Jain, A. Hans, M. Shu, A. Tomar, T. Goldstein, and A. Bhatele, “Democratizing ai: Open-source scalable llm training on gpu-based supercomputers,” in Proceedings of the International Conference for High...

    2024 arXiv

  26. [2021]

    Available: https://arxiv.org/abs/2009.03300

    [Online]. Available: https://arxiv.org/abs/2009.03300

    2009 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.