REVIEW 12 cited by
Zero Bubble Pipeline Parallelism
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work, we introduce a scheduling strategy that, to our knowledge, is the first to successfully achieve zero pipeline bubbles under synchronous training semantics. The key idea behind this improvement is to split the backward computation into two parts, one that computes gradient for the input and another that computes for the parameters. Based on this idea, we handcraft novel pipeline schedules that significantly outperform the baseline methods. We further develop an algorithm that automatically finds an optimal schedule based on specific model configuration and memory limit. Additionally, to truly achieve zero bubble, we introduce a novel technique to bypass synchronizations during the optimizer step. Experimental evaluations show that our method outperforms the 1F1B schedule up to 23% in throughput under a similar memory limit. This number can be further pushed to 31% when the memory constraint is relaxed. We believe our results mark a major step forward in harnessing the true potential of pipeline parallelism. We open sourced our implementation based on the popular Megatron-LM repository on https://github.com/sail-sg/zero-bubble-pipeline-parallelism.
Forward citations
Cited by 12 Pith papers
-
Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
ADAPTRA adapts the pipeline schedule and offloads inter-stage communication to CPU-side RDMA, cutting iteration time by 1.2-3.5x when network links slow down.
-
Opus: Photonic Rail-Optimized Fabric in ML Datacenters
Opus time-multiplexes a single photonic rail fabric across parallelism phases in ML training, achieving up to 23x network power reduction and 4x cost savings at under 6.7% training overhead in simulation.
-
MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster
MindSpeed RL uses a distributed transfer dock and an allgather-swap method to raise RL training throughput on Ascend NPUs by 1.42 to 3.97 times over two open baselines.
-
Photonic Rails in ML Datacenters
A photonic rail design that reconfigures optical circuits between parallelism phases within a training job can emulate electrical rails with about 70% cost and 96% power savings and a few percent iteration-time overhe...
-
Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
Thunder-Tok, a Korean tokenizer with grammar-based pre-tokenization and branching-entropy vocabulary selection, cuts tokens per word by about 10% versus BPE while keeping downstream performance comparable.
-
Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
Hetu v2 introduces hierarchical and heterogeneous SPMD annotations plus graph specialization and dynamic graph switching, enabling efficient LLM training on heterogeneous, unstable, and mixed-length workloads.
-
Echo: Simulating Distributed Training At Scale
Echo predicts distributed-training step time with about 8% error by tracing workloads on one GPU, modeling NCCL communication analytically, and learning overlap-induced kernel slowdowns.
-
DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks
DeepCEE groups heterogeneous GPUs by network and compute speed, schedules a compact zero-bubble pipeline across regions, and adapts micro-batch sizes to network fluctuations, reporting 1.3-2.8x higher training through...
-
Nesterov Method for Asynchronous Pipeline Parallel Optimization
A discounted Nesterov look-ahead update, implemented with NAdam, is claimed to converge sublinearly under fixed gradient delay and to beat synchronous pipeline baselines in language modeling.
-
Automatically Planning Optimal Parallel Strategy for Large Language Models
A training-time simulator and pruned search space choose the fastest 3D parallel strategy, including micro-batch size, for LLM training on clusters.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
-
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
A survey of DeepSeek's V3 and R1 models covering MLA, MoE, MTP, GRPO, and training engineering, with no new experimental results.
Discussion (0). Continue with ORCID to comment.