Pith. sign in

REVIEW 2 cited by

torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2004.09910 v1 pith:5AXI7AXP submitted 2020-04-21 cs.DC cs.LG

Chiheon Kim , Heungsub Lee , Myungryong Jeong , Woonhyuk Baek , Boogeon Yoon , Ildoo Kim , Sungbin Lim , Sungwoong Kim This is my paper
classification cs.DCcs.LG
keywords libraryparallelismpipelinedesignenvironmentpytorchtorchgpipeamoebanet-d
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We design and implement a ready-to-use library in PyTorch for performing micro-batch pipeline parallelism with checkpointing proposed by GPipe (Huang et al., 2019). In particular, we develop a set of design components to enable pipeline-parallel gradient computation in PyTorch's define-by-run and eager execution environment. We show that each component is necessary to fully benefit from pipeline parallelism in such environment, and demonstrate the efficiency of the library by applying it to various network architectures including AmoebaNet-D and U-Net. Our library is available at https://github.com/kakaobrain/torchgpipe .

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distributed Deep Learning using Stochastic Gradient Staleness

    cs.LG 2025-09 reject novelty 5.0 of 10

    A hybrid data- and model-parallel training scheme using stale gradients and consensus averaging, claimed to converge to critical points and speed up ResNet-20 training on CIFAR-10.

  2. FlashDP: Private Training Large Language Models with Efficient DP-SGD

    cs.LG 2025-07 conditional novelty 5.0 of 10

    FlashDP fuses per-sample gradient computation, norm calculation, clipping, and noise addition into a cache-friendly block-wise all-reduce workflow that avoids explicit per-sample gradient storage and redundant recomputation.

Pith tools