Pith. sign in

REVIEW 15 cited by

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06511 v3 pith:YXYZUDTE submitted 2024-10-09 cs.CL cs.AIcs.DCcs.LG

Wanchao Liang , Tianyu Liu , Less Wright , Will Constable , Andrew Gu , Chien-Chin Huang , Iris Zhang , Wei Feng
show 5 more authors
Howard Huang Junjie Wang Sanket Purandare Gokul Nadathur Stratos Idreos
This is my paper · ORCID
classification cs.CLcs.AIcs.DCcs.LG
keywords trainingllamatorchtitanparallelismscalellmsstate-of-the-arttechniques
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The development of large language models (LLMs) has been instrumental in advancing state-of-the-art natural language processing applications. Training LLMs with billions of parameters and trillions of tokens require sophisticated distributed systems that enable composing and comparing several state-of-the-art techniques in order to efficiently scale across thousands of accelerators. However, existing solutions are complex, scattered across multiple libraries/repositories, lack interoperability, and are cumbersome to maintain. Thus, curating and empirically comparing training recipes require non-trivial engineering effort. This paper introduces TorchTitan, an open-source, PyTorch-native distributed training system that unifies state-of-the-art techniques, streamlining integration and reducing overhead. TorchTitan enables 3D parallelism in a modular manner with elastic scaling, providing comprehensive logging, checkpointing, and debugging tools for production-ready training. It also incorporates hardware-software co-designed solutions, leveraging features like Float8 training and SymmetricMemory. As a flexible test bed, TorchTitan facilitates custom recipe curation and comparison, allowing us to develop optimized training recipes for Llama 3.1 and provide guidance on selecting techniques for maximum efficiency based on our experiences. We thoroughly assess TorchTitan on the Llama 3.1 family of LLMs, spanning 8 billion to 405 billion parameters, and showcase its exceptional performance, modular composability, and elastic scalability. By stacking training optimizations, we demonstrate accelerations of 65.08% with 1D parallelism at the 128-GPU scale (Llama 3.1 8B), an additional 12.59% with 2D parallelism at the 256-GPU scale (Llama 3.1 70B), and an additional 30% with 3D parallelism at the 512-GPU scale (Llama 3.1 405B) on NVIDIA H100 GPUs over optimized baselines.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Rayleigh criterion for mechanical instability: inducing activity by chemo-mechanical coupling

    cond-mat.stat-mech 2026-05 unverdicted novelty 7.0 of 10

    Rayleigh-like phase criteria between entropic and frenetic contributions predict when chemical driving of a slow Newtonian probe yields sustained active or rotational motion.

  2. Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

    cs.LG 2025-06 reject novelty 7.0 of 10

    Constraining transformer projection weights to a shared low-rank subspace reportedly enables near-lossless compression of pipeline-parallel communication, matching centralized convergence at 80Mbps bandwidth.

  3. Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation

    cs.DC 2025-04 conditional novelty 7.0 of 10

    ADAPTRA adapts the pipeline schedule and offloads inter-stage communication to CPU-side RDMA, cutting iteration time by 1.2-3.5x when network links slow down.

  4. Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Under a fixed model-FLOPs budget, the fitted loss for sparse MoE models decreases monotonically with sparsity, so the optimum sits at the boundary; an interior sparsity optimum appears only when hardware-deliverable F...

  5. StrataCL: Fabric-Native Communication Library for Production Supernodes

    cs.DC 2026-07 conditional novelty 6.0 of 10

    On Huawei CloudMatrix384, StrataCL's registration-on-allocation and fabric-native operators raise collective bus bandwidth up to 1.6x and LLM inference throughput by 1.9x versus HCCL baselines.

  6. MatrixFSDP: communication-free matrix optimizers under ZeRO-3 parameter sharding

    cs.DC 2026-07 conditional novelty 6.0 of 10

    MatrixFSDP places whole 2D weight matrices on single ZeRO-3 owner ranks so matrix optimizers run locally without optimizer-step collectives, preserving ZeRO-3 memory while achieving up to 54.6x optimizer-step speedup ...

  7. Opus: Photonic Rail-Optimized Fabric in ML Datacenters

    cs.NI 2026-02 conditional novelty 6.0 of 10

    Opus time-multiplexes a single photonic rail fabric across parallelism phases in ML training, achieving up to 23x network power reduction and 4x cost savings at under 6.7% training overhead in simulation.

  8. Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters

    cs.DC 2025-07 conditional novelty 6.0 of 10

    Zorse integrates interleaved pipeline parallelism, ZeRO-2 data parallelism, and CPU offloading to accelerate LLM training on heterogeneous GPU clusters by up to 4x.

  9. Photonic Rails in ML Datacenters

    cs.NI 2025-07 conditional novelty 6.0 of 10

    A photonic rail design that reconfigures optical circuits between parallelism phases within a training job can emulate electrical rails with about 70% cost and 96% power savings and a few percent iteration-time overhe...

  10. Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage

    cs.DC 2025-06 conditional novelty 6.0 of 10

    TERAIO speeds up LLM training by offloading inactive tensors to SSDs with a lifetime-aware, GPUDirect Storage based migration plan, reaching 1.47x over ZeRO-Offload/Infinity and 80.7% of ideal.

  11. Scaling Deep Learning Training with MPMD Pipeline Parallelism

    cs.DC 2024-12 conditional novelty 6.0 of 10

    JaxPP introduces a user-defined MPMD pipeline schedule API on top of JAX/GSPMD and reports throughput gains up to 1.11x over SPMD training on H100 clusters.

  12. Motif 3: Technical Report

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A 314B-parameter mixture-of-experts language model with grouped differential latent attention and multi-teacher distillation reports competitive results on agentic, coding, and reasoning benchmarks.

  13. TorchAO: PyTorch-Native Training-to-Serving Model Optimization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    TorchAO delivers a PyTorch-native, end-to-end workflow for FP8 training, QAT, PTQ, and sparsity, with benchmarked speedups and production use in quantized Llama releases.

  14. Chain-of-Model Learning for Language Model

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A nested Transformer with causally ordered hidden chains offers multiple sub-model sizes, chain-based expansion, and KV-cache sharing for faster prefilling.

  15. Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PRIOR reweights the next-token prediction loss in vision-language pretraining by 1 minus the probability assigned by a text-only reference LLM, and reports consistent benchmark improvements over standard NTP.

Pith tools