REVIEW 3 cited by
Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
read the original abstract
Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.
Forward citations
Cited by 3 Pith papers
-
Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
A tile-level signaling and scheduling co-design overlaps MoE expert GEMM with the return all-to-all, achieving up to 2.64x end-to-end speedup on 4 A100s.
-
Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training
Kareus jointly tunes kernel timing, GPU core allocation, and frequency to expand the time–energy tradeoff frontier for training large language models.
-
Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap
Finer-grain, DMA-offloaded compute-communication overlap with schedule heuristics yields up to 1.6x speedups for distributed ML on full-mesh GPU systems.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.