Pith. sign in

REVIEW 4 cited by

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.14883 v3 pith:STSXVC2Y submitted 2021-10-28 cs.LG cs.AIcs.CLcs.CVcs.DC

Shenggui Li , Hongxin Liu , Zhengda Bian , Jiarui Fang , Haichen Huang , Yuliang Liu , Boxiang Wang , Yang You This is my paper
classification cs.LGcs.AIcs.CLcs.CVcs.DC
keywords trainingparallelcolossal-aideeplearningsystemlarge-scalemethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice for choosing the optimal parallel strategy is still lacking, since it requires domain expertise in both deep learning and parallel computing. The Colossal-AI system addressed the above challenge by introducing a unified interface to scale your sequential code of model training to distributed environments. It supports parallel training methods such as data, pipeline, tensor, and sequence parallelism, as well as heterogeneous training methods integrated with zero redundancy optimizer. Compared to the baseline system, Colossal-AI can achieve up to 2.76 times training speedup on large-scale models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Scaling Data-Constrained Language Models

    cs.CL 2023-05 conditional novelty 6.0 of 10

    Repeating training data up to 4 epochs yields negligible loss increase versus unique data for fixed compute, and a new scaling law accounts for the decaying value of repeated tokens and excess parameters.

  2. JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models

    cs.DC 2026-07 conditional novelty 5.0 of 10

    A service-oriented multi-tenant architecture with schema-compatible group batching reduces aggregate GPU time for VLA post-training by about 28% in simulation.

  3. A Survey of Large Language Models

    cs.CL 2023-03 accept novelty 3.0 of 10

    This survey reviews the background, key techniques, and evaluation methods for large language models, emphasizing emergent abilities that appear at large scales.

  4. A Comprehensive Overview of Large Language Models

    cs.CL 2023-07 unverdicted novelty 2.0 of 10

    A survey paper providing an overview of Large Language Models, their background, and recent advances in the field.

Pith tools