REVIEW 6 cited by
Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelis
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Deep neural network (DNN) training continues to scale rapidly in terms of model size, data volume, and sequence length, to the point where multiple machines are required to fit large models for training. Different distributed and parallel training strategies have been developed to support large-scale DNN training by partitioning the training state across GPUs. However, existing DNN training systems provide very limited support for reconfiguring parallelism strategies in the middle of the training via checkpointing. This limitation arises because distributed checkpoints are tightly coupled to specific model parallelism and hardware configurations, preventing large-scale training jobs from efficiently adapting to hardware failures or resource elasticity. This paper presents Universal Checkpointing (UCP), a novel checkpointing system that enables flexible and efficient DNN training with reconfigurable parallelism. UCP overcomes challenges in existing systems by decoupling checkpoint structure from parallel training strategies and hardware configurations. In addition, we present a pattern-based reconfiguration pipeline that enables automatic, flexible, and efficient mapping of checkpoint state to various parallelism strategies. Evaluation on a range of DNN models, including state-of-the-art dense and sparse LLMs, shows that UCP enables reconfiguration for a broader set of widely used parallelism strategies than existing solutions while adding negligible reconfiguration cost. UCP has been successfully employed in real LLM training workloads, greatly enhancing their flexibility and resilience to dynamic hardware environments.
Forward citations
Cited by 6 Pith papers
-
TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure
TensorCast packages tensor lifecycle management into a programmable service layer and reports it can match specialized LLM systems while enabling new cross-component optimization policies.
-
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
PHOENIX recovers permanent node failures in LLM training via hot-swapping of spares using zero-overhead per-step in-memory optimizer-state replication, finishing recovery in under 40 s on up to 512 GPUs.
-
Cloud abstractions for AI workloads
HarmonAIze proposes new cloud abstractions, split into micro- and macro-level control, to enable tenant-provider cooperation for more efficient and resilient AI workloads.
-
FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs
FlashRecovery restores failed large-scale LLM training from data-parallel replicas in about 150 seconds on 4,800 devices, removing the need for periodic checkpointing.
-
A Theoretical Framework for Virtual Power Plant Integration with Gigawatt-Scale AI Data Centers: Multi-Timescale Control and Stability Analysis
The paper sketches a hierarchical VPP control architecture for gigawatt AI data centers, but its key stability and performance results depend on unverified assumptions and a fitted constant.
-
YuLan-Mini: An Open Data-efficient Language Model
A 2.42B-parameter base model trained on 1.08T tokens matches or beats several industry baselines trained on 7T to 18T tokens across math, code, and general benchmarks.
Discussion (0). Continue with ORCID to comment.