A topology-aware three-level partitioning of weights, gradients, and optimizer states speeds up ZeRO++ LLM training on Frontier by roughly 1.7x per GPU for 20B models with 0.94 scaling efficiency.
The mvapich project: Transforming research into high-performance mpi library for hpc community,
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.DC 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning
A topology-aware three-level partitioning of weights, gradients, and optimizer states speeds up ZeRO++ LLM training on Frontier by roughly 1.7x per GPU for 20B models with 0.94 scaling efficiency.