REVIEW 6 cited by
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
read the original abstract
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and significant computing time, posing major challenges in terms of scalability, efficiency, and reliability. This survey explores recent advancements in training systems for LLMs, including innovations in training infrastructure with AI accelerators, networking, storage, and scheduling. Additionally, the survey covers parallelism strategies, as well as optimizations for computation, communication, and memory in distributed LLM training. It also includes approaches of maintaining system reliability over extended training periods. By examining current innovations and future directions, this survey aims to provide valuable insights towards improving LLM training systems and tackling ongoing challenges. Furthermore, traditional digital circuit-based computing systems face significant constraints in meeting the computational demands of LLMs, highlighting the need for innovative solutions such as optical computing and optical networks.
Forward citations
Cited by 6 Pith papers
-
MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall
MLP-Offload accelerates LLM pre-training on memory-constrained GPUs by mixing local NVMe and remote PFS offloading with cache-aware subgroup reordering, achieving up to 2.5x faster iterations than DeepSpeed ZeRO-3.
-
Design-CP: Context Parallelism for Design of Protein Nanoparticles
Context-parallel inference for RFdiffusion 3 enables end-to-end all-atom design of large symmetric protein nanoparticles on multi-GPU hardware without retraining.
-
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
CommFuse eliminates tail latency in communication-computation overlap for distributed LLM training by decomposing collective operations into P2P communications and fusing them with fine-grained computation scheduling.
-
ChipLight: Cross-Layer Optimization of Chiplet Design with Optical Interconnects for LLM Training
ChipLight is a multi-objective optimization framework that co-designs chiplet hardware, training parallelism, and optical networks to improve efficiency in distributed LLM training clusters.
-
Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference
A hybrid JIT-CUDA Graph framework reduces TTFT by up to 66% and P99 latency versus TensorRT-LLM for single-GPU LLaMA-2 7B inference on short prompts.
-
Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI
A framework for runtime re-splitting and re-placement of foundation model layers across edge nodes is proposed, but its claimed latency gains are inherited from prior work rather than measured.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.