REVIEW 5 cited by
Maximizing Parallelism in Distributed Training for Huge Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The recent Natural Language Processing techniques have been refreshing the state-of-the-art performance at an incredible speed. Training huge language models is therefore an imperative demand in both industry and academy. However, huge language models impose challenges to both hardware and software. Graphical processing units (GPUs) are iterated frequently to meet the exploding demand, and a variety of ASICs like TPUs are spawned. However, there is still a tension between the fast growth of the extremely huge models and the fact that Moore's law is approaching the end. To this end, many model parallelism techniques are proposed to distribute the model parameters to multiple devices, so as to alleviate the tension on both memory and computation. Our work is the first to introduce a 3-dimensional model parallelism for expediting huge language models. By reaching a perfect load balance, our approach presents smaller memory and communication cost than existing state-of-the-art 1-D and 2-D model parallelism. Our experiments on 64 TACC's V100 GPUs show that our 3-D parallelism outperforms the 1-D and 2-D parallelism with 2.32x and 1.57x speedup, respectively.
Forward citations
Cited by 5 Pith papers
-
TrainVerify: Equivalence-Based Verification for Distributed LLM Training
TrainVerify verifies distributed LLM training execution plans against logical model definitions using symbolic dataflow graphs, shape reduction, and staged SMT solving, scaling to 671B-parameter models.
-
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.
-
DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks
DeepCEE groups heterogeneous GPUs by network and compute speed, schedules a compact zero-bubble pipeline across regions, and adapts micro-batch sizes to network fluctuations, reporting 1.3-2.8x higher training through...
-
Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models
Pro-Prophet balances MoE training load by moving each expert to only the devices that need it and by pre-scheduling those moves to hide communication behind computation.
-
You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models
D2FT dynamically schedules attention subnets to full, forward-only, or skipped operations during fine-tuning, cutting compute 40% and communication 50% with 1-2% top-1 accuracy loss on three vision datasets, plus a Lo...
Discussion (0). Continue with ORCID to comment.