Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Beyond roughly 128 GPUs, model parallelism outperforms data-parallel sharding for LLM training.

desk verdict Solid, wide-ranging FSDP scaling study with a real finding; the headline claim of 'properly optimized' overreaches past the untested HSDP family. read the letter →

arxiv 2411.13055 v2 pith:V4ACOPCX submitted 2024-11-20 cs.LG cs.DC

Jared Fernandez , Luca Wehrstedt , Leonid Shamis , Mostafa Elhoushi , Kalyan Saladi , Yonatan Bisk , Emma Strubell , Jacob Kahn This is my paper · ORCID
classification cs.LGcs.DC
keywords distributedtrainingFullyShardedDataParallelismmodelcommunicationoverheadscalinglawspowerefficiencyLLMthroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large-scale distributed training of language models has two under-appreciated properties: past a scale of roughly 128 GPUs, the communication overhead of fully sharded data parallelism (FSDP) is large enough that adding tensor or pipeline parallelism—traditionally viewed as a utilization tax—actually increases throughput; and beyond a few hundred GPUs, additional accelerators deliver shrinking per-GPU returns even when the parallelization strategy is chosen well, because per-GPU power draw stays roughly constant while per-GPU throughput falls. The authors support this with measurements across 1B to 70B parameter models on up to 2048 H100 and A100 GPUs. If correct, the results imply that the common practice of sharding parameters and scaling out data-parallel replicas is compute- and energy-inefficient at pretraining scale, and that scaling laws built on FLOP counts alone will overestimate real throughput.

What carries the argument

The load-bearing object is the exposed-communication mechanism of Fully Sharded Data Parallelism: parameter materialization and gradient reduction go through NCCL AllGather and ReduceScatter collectives, which use ring algorithms whose latency scales with node count (measured in Figure 2), unlike the tree-based AllReduce used by standard data parallelism and tensor parallelism. Because communication-bound kernels cannot be hidden by computation once the per-device compute time is small relative to collective latency, exposed communication grows and hardware utilization collapses. Model parallelism is the lever that shrinks the FSDP collective's world size, trading AllReduce and pipeline-bubble costs for reduced AllGather/ReduceScatter volume, which the paper measures as lower exposed communication and higher global throughput.

What would settle it

Train Llama-7B at 2048 H100 GPUs with hybrid sharded data parallelism (sharding only within a node) or with ZeRO-3 using node-local sharding, under the same fixed global batch size and measured with the paper's methodology, and compare per-GPU throughput and MFU to the paper's best tensor-parallel configuration. If per-GPU throughput fails to decline by roughly a third relative to 128 GPUs, or MFU stays above 40%, the claimed diminishing returns under proper optimization would not hold.

Watch

Extended reading notes

Core claim

The central discovery is that communication boundedness, not compute, governs large-scale FSDP training. The AllGather and ReduceScatter collectives used to rematerialize and update sharded parameters are ring-based and their latency grows with world size, so as nodes are added the ratio of communication to computation rises; beyond 128 H100 GPUs for a 7B model, exposed communication is unavoidable, hardware utilization (MFU) falls, and throughput per device drops by 37.22% when scaling from 128 to 2048 GPUs. Because GPU power draw declines only slightly (5.87%) while throughput falls, tokens per watt decline markedly. The paper shows that moderate model parallelism (tensor or pipeline degrees 2–4) shrinks the data-parallel group sizes for FSDP collectives and thereby reduces exposed communication, improving throughput, MFU, and power efficiency relative to the data-parallel baseline—including a +52.60% words-per-second gain at 2048 GPUs. It also shows that faster compute relative to interconnect (H100 vs A100) worsens communication boundedness, and that scaling model size increases both compute and communication jointly.

Load-bearing premise

The claim that returns diminish 'even when hardware and parallelization strategies are properly optimized' assumes the tested sweep over tensor and pipeline parallelism (degrees up to 16 on fixed node counts) covers the strategies a truly optimized system would use; the paper does not test hybrid sharded data parallelism, ZeRO-3 with different sharding policies, or newer fabric topologies, any of which could in principle flatten the diminishing-returns curve.

Editorial extensions

If this is right

  • Practitioners training 7B-class models on clusters above roughly 128 GPUs should add tensor parallelism at degree 2 or 4 instead of relying on FSDP data-parallel replicas, because it reduces exposed communication and raises global throughput.
  • Throughput-based scaling laws that assume linear gains with added accelerators will overestimate output at pretraining scale; compute-optimal scaling should incorporate communication cost.
  • Energy per token rises with cluster size, so the environmental cost of a training run is not proportional to FLOPs but to a less favorable curve.
  • Faster GPU compute alone will not fix the problem; interconnect and memory bandwidth improvements are needed to prevent training from becoming more communication-bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The untested strategies the paper defers, especially hybrid sharded data parallelism that keeps AllGather inside a node, could plausibly shift the crossover point where model parallelism becomes preferable; the reported curves may be an upper bound on what current FSDP-like training can achieve.
  • The diminishing-returns pattern likely extends to TPU-based training and to inference-time parallel decoding, because the same ring-collective latency growth applies whenever sharded parameters must be gathered across many devices.
  • A testable design implication is that node-local sharding with inter-node only AllReduce—the hybrid pattern—should recover much of the lost throughput, which would directly quantify how much of the reported 37% drop is due to the ring AllGather world size.
  • Scaling-law papers that ignore communication will increasingly mispredict time-to-train as clusters grow; replacing FLOP count with a communication-aware cost measure is a tractable near-term correction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents an extensive empirical study of distributed training of Llama-2-style LLMs (1B to 70B parameters) on clusters ranging from 8 to 2048 H100/A100 GPUs, using PyTorch FSDP combined with optional tensor and pipeline parallelism. The central measurements are global and per-device throughput, exposed communication (from profiled NCCL/CUDA kernel times), MFU, and GPU power draw. The paper reports three main findings: (1) FSDP's AllGather/ReduceScatter communication overhead grows with data-parallel world size, making model parallelism (tensor or pipeline, at degrees 2--4) beneficial at scales beyond roughly 128 GPUs; (2) global throughput increases sublinearly with added accelerators while per-GPU power draw remains nearly flat, so tokens-per-watt and MFU decline with scale; and (3) the optimal configurations for sharded training differ from earlier recommendations that model parallelism be avoided. The abstract and conclusion generalize these results to claim that scaling accelerators yields diminishing returns 'even when hardware and parallelization strategies are properly optimized.'

Significance. If the conclusions hold, they provide actionable guidance for training large models at scale: flat FSDP becomes communication-bound above a few hundred GPUs, and moderate tensor/pipeline parallelism is preferable; they also highlight an energy-efficiency penalty of scale that is largely absent from existing scaling-law discussions. The paper's strengths are the unusual scale of the measurements (up to 2048 H100s), the breadth across model sizes and hardware generations, the detailed kernel-level accounting of exposed communication, and the fact that the trends are direct observations rather than outputs of a fitted parametric model. The central caveat is that the 'properly optimized' claim is conditional on a limited strategy family; this affects the generality of the diminishing-returns conclusion, not the validity of the measurements themselves.

major comments (3)
  1. [Abstract, §4.3, Appendix A] The abstract and Section 5 assert diminishing returns 'even when hardware and parallelization strategies are properly optimized,' but the optimization search in Section 4.3 is restricted to tensor and pipeline parallelism of degree at most 16 combined with a flat FSDP sharding group. Appendix A explicitly defers Hybrid Sharded Data Parallelism (HSDP) and other hierarchical sharding strategies to future work. This is not a peripheral omission: Sections 2.2 and Figure 2 identify AllGather/ReduceScatter world size as the scaling bottleneck, and HSDP is precisely a way to keep those collectives inside 8-GPU nodes. If HSDP (or a similar hierarchical scheme) performs sufficiently better at 2048 GPUs, the reported diminishing-returns curve and the +52.60% improvement from tensor parallelism may not represent the 'properly optimized' configuration. The measurements remain valid as a characterization of flat FSDP, but the general claim should be scoped to the considered parallelism family, or HSDP experiments should be added at the largest scales.
  2. [§3 (Metrics), §4.1, §4.3] Section 3 states that metrics are aggregated over 50 training iterations after 10 warmup iterations, but this appears to be a single run and no run-to-run variance or confidence intervals are reported. Because the central quantitative claims (e.g., the 37.22% per-device throughput decrease from 128 to 2048 GPUs in Section 4.1 and the +52.60% WPS improvement from tensor parallelism in Section 5) are used to support the diminishing-returns conclusion, the absence of variance information makes it impossible to judge whether the differences between configurations are statistically significant. Please report standard deviations over at least a few repeated runs (or justify why the environment is deterministic and demonstrate this with measurements).
  3. [§4.2, Figure 5] Section 4.2 and Figure 5 describe strong scaling with a 'fixed global batch size of 32 examples' across 2 to 32 nodes, but with 256 GPUs and global batch 32, a pure data-parallel configuration would require a sub-unit per-device batch. Presumably the optimal configurations use enough model parallelism to keep the data-parallel degree no larger than 32, but the text does not state how the configurations were re-searched per node count. The caption also says the optimal strategy is 'determined by the experimental results displayed in Figure 6,' which is for a different setup (256 GPUs, global batch 512). Please clarify the selection procedure and the feasibility of the global batch at each scale; this is load-bearing for the strong-scaling results.
minor comments (5)
  1. [Introduction, Related Work] There are several typos and inconsistencies: 'langauge' in the introduction, 'Zero-3' instead of 'ZeRO-3' in Section 6, and inconsistent capitalization of 'FLOPS'/'FLOPs' throughout.
  2. [§2.2, Figure 2, Appendix B] The units for AllReduce/AllGather bandwidth in Figure 2 are reported as GB/s, while Appendix B refers to per-node InfiniBand bandwidth in GB/s; please clarify whether these are GB/s or Gb/s, and whether the 'favorable scaling' of AllReduce bandwidth is specific to the tree algorithm or a general statement.
  3. [§4.4] The text says 'nearly 20% decrease in hardware utilization' when comparing A100 to H100 (59.67% to 40.77%). This is a decrease of about 19 percentage points but roughly 32% relative; please report the relative and absolute changes explicitly to avoid ambiguity.
  4. [§3 (Power utilization)] Power efficiency is based on NVML GPU power draw only; the paper should state explicitly that total system power (including CPU, memory, and network) is not included, so the energy-efficiency claims apply to GPU power consumption.
  5. [Appendix F, Figure 13] In Figure 13, the y-axis for 'Exposed Communication' is labeled 'Time (microseconds)' but the values are on the order of 1e6, which is inconsistent; the unit is likely milliseconds. Please fix the axis label.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the scaling and parallelization conclusions are direct measurements, not fitted predictions or self-citation reductions.

full rationale

The paper's central claims are empirical rather than derivational: throughput, MFU, power draw, and NCCL kernel times are measured across hardware scales, model sizes, and parallelization strategies and reported directly (Figures 3, 5, 6, 8). The conclusion that FSDP becomes communication-bound at scale follows from measured AllGather/ReduceScatter scaling and measured exposed communication, not from a parameter fitted to the target claim. Selecting the best observed tensor/pipeline configuration at each scale in Sections 4.2-4.3 is ordinary empirical search; the diminishing-returns curve is the measured envelope of searched strategies, not a quantity defined in terms of itself. The only self-references (Fernandez et al. 2023; Strubell et al. 2019) appear in related-work and environmental-impact context and are not load-bearing for the scaling results. Appendix A explicitly defers Hybrid Sharded Data Parallelism and variable fabric speeds to future work, which is a genuine external-validity caveat on the phrase 'even when hardware and parallelization strategies are properly optimized'; the conclusion is conditioned on the tested strategy family. That incompleteness is a correctness risk, not circularity, because no step in the paper reduces by construction to its own input.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper's conclusions rest entirely on the validity and representativeness of its measurements. No theoretical derivation is supplied, so the ledger contains only domain assumptions about hardware representativeness, profiler accuracy, and the completeness of the parallelization search. There are no fitted free parameters and no invented entities.

assumptions (5)
  • domain assumption The measured clusters and software stack are representative of typical datacenter training platforms.
    All measurements use Meta's internal DGX clusters with NVLink and InfiniBand and PyTorch FSDPv2; the paper generalizes to 'large-scale distributed training' broadly, but this is not validated on other fabrics or frameworks.
  • ad hoc to paper Fifty measured training iterations after ten warmup iterations yield steady-state performance metrics.
    The choice of 50 iterations is a practical compromise and is not justified by variance analysis; longer runs could change average throughput or power draw.
  • domain assumption The profiler-based separation of exposed versus overlapped communication correctly captures hardware stalls.
    The paper relies on PyTorch profiler, Kineto, and PerfettoSQL to attribute CUDA and NCCL kernel times; any inaccuracies in overlap detection affect the central 'exposed communication grows with scale' story.
  • domain assumption Llama-2 is a representative decoder-only architecture for scaling conclusions.
    The paper only studies Llama-2 models and states that extension to other architectures is future work.
  • ad hoc to paper The parallelization search space considered, tensor and pipeline parallelism up to degree 16, contains the optimal configurations for the studied workloads.
    HSDP, context parallelism (briefly tested in Appendix E), and other hybrid strategies are not part of the main search; the 'optimized' configurations in Figures 5 and 11 are only optimal within this restricted grid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training." pith.science (2026). https://pith.science/paper/V4ACOPCX

@misc{pith2026241113055,
  author       = {Pith},
  title        = {Pith review of: Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4ACOPCX}},
  note         = {Machine review of arXiv:2411.13055}
}
read the original abstract

Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To develop the exceedingly large networks required in modern applications, such as large language models (LLMs), model training is distributed across tens of thousands of hardware accelerators (e.g. GPUs), requiring orchestration of computation and communication across large computing clusters. In this work, we demonstrate that careful consideration of hardware configuration and parallelization strategy is critical for effective (i.e. compute- and cost-efficient) scaling of model size, training data, and total computation. We conduct an extensive empirical study of the performance of large-scale LLM training workloads across model size, hardware configurations, and distributed parallelization strategies. We demonstrate that: (1) beyond certain scales, overhead incurred from certain distributed communication strategies leads parallelization strategies previously thought to be sub-optimal in fact become preferable; and (2) scaling the total number of accelerators for large model training quickly yields diminishing returns even when hardware and parallelization strategies are properly optimized, implying poor marginal performance per additional unit of power or GPU-hour.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Under a fixed model-FLOPs budget, the fitted loss for sparse MoE models decreases monotonically with sparsity, so the optimum sits at the boundary; an interior sparsity optimum appears only when hardware-deliverable F...

  2. Evaluating MFU as a Proxy for GPU Power for Energy-Aware Simulation of LLM Training

    cs.PF 2026-08 conditional novelty 6.0 of 10

    MFU is a valid linear predictor of GPU power in compute-bound LLM training, with per-(GPU, dtype, batch) fits reaching 1% error, matching measurement noise.

  3. Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A simulation framework couples an LLM inference simulator with a GPU power model and an energy-grid co-simulator to estimate energy and carbon emissions across deployment configurations.

  4. Estudio de la eficiencia en la escalabilidad de GPUs para el entrenamiento de Inteligencia Artificial

    cs.LG 2025-09 conditional novelty 3.0 of 10

    Using MLPerf v4.1 data, the paper shows that per-GPU efficiency declines as GPU count grows, and it identifies intermediate configurations that balance training speed and resource use.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages · cited by 4 Pith papers

  1. [5]

    The framework tax: Disparities between inference efficiency in nlp research and deployment

    Jared Fernandez, Jacob Kahn, Clara Na, Yonatan Bisk, and Emma Strubell. The framework tax: Disparities between inference efficiency in nlp research and deployment. InThe 2023 Conference on Empirical Methods in Natural Language Processing,

    work page 2023

  2. [6]

    ISBN 9781450357999

    Association for Computing Machinery. ISBN 9781450357999. doi: 10.1145/3210377.3210394. URL https://doi.org/10.1145/3210377.3210394. IBM Granite Team. Granite 3.0 language models,

    arXiv

  3. [9]

    John L Hennessy and David A Patterson.Computer architecture: a quantitative approach

    URLhttps://arxiv.org/abs/1806.03377. John L Hennessy and David A Patterson.Computer architecture: a quantitative approach. Morgan kaufmann,

    arXiv

  4. [12]

    Sasha Luccioni, Yacine Jernite, and Emma Strubell

    URLhttps://arxiv.org/abs/2302.08476. Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 85–99,

    arXiv 2024

  5. [13]

    Fully sharded data parallel: faster ai training with fewer gpus — engineering.fb.com.https://engineering.fb.com/2021/07/15/open-source/fsdp/,

    Myle Ott, Sam Shleifer, Min Xu, Priya Goyal, Quentin Duval, and Vittorio Caggiano. Fully sharded data parallel: faster ai training with fewer gpus — engineering.fb.com.https://engineering.fb.com/2021/07/15/open-source/fsdp/,

    work page 2021

  6. [15]

    Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146,

    Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146,

    arXiv

  7. [16]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    arXiv

  8. [17]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro

    URLhttps://arxiv.org/abs/1811.02084. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron- lm: Training multi-billion parameter language models using model parallelism.ArXiv, abs/1909.08053,

    arXiv 1909

Show all 25 references
  1. [18]

    Sebastian U Stich

    URL https://api.semanticscholar.org/CorpusID:202660670. Sebastian U Stich. Local sgd converges fast and communicates little.arXiv preprint arXiv:1805.09767,

    arXiv

  2. [19]

    doi: 10.18653/v1/P19-1355

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1355. URL https://aclanthology.org/P19-1355. Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs mod...

    doi

  3. [20]

    Accessed: 2023-05-05

    URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05. Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fengqi Qiao, Piotr Majcher, Paul Springer, ...

    2023

  4. [21]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al

    URLhttps://github.com/NVIDIA/ cutlass. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2...

    arXiv

  5. [22]

    Context parallelism for scalable million-token inference.arXiv preprint arXiv:2411.01783,

    Amy Yang, Jingyi Yang, Aya Ibrahim, Xinfeng Xie, Bangsheng Tang, Grigory Sizov, Jeremy Reizenstein, Jongsoo Park, and Jianyu Huang. Context parallelism for scalable million-token inference.arXiv preprint arXiv:2411.01783,

    arXiv

  6. [23]

    Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism

    14 Tailing Yuan, Yuliang Liu, Xucheng Ye, Shenglong Zhang, Jianchao Tan, Bin Chen, Chengru Song, and Di Zhang. Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism. In 2024 USENIX Annual Technical Confer...

    2024

  7. [24]

    For our primary experiments, we trained models using PyTorch 2.3.1 built with CUDA 12.1, with attention implementation provided by XFormers 0.27

    and xFormers (Lefaudeux et al., 2022). For our primary experiments, we trained models using PyTorch 2.3.1 built with CUDA 12.1, with attention implementation provided by XFormers 0.27. We utilize PyTorch FSDPv2 with prefetch of subsequent layers enabled. For the A100 and H100 ...

    2022

  8. [25]

    Nodes within the V100 cluster consist of 8-GPU setups connected with first-generation NVLink in a Hybrid Cube Mesh (HCM) topology

    attention kernels on Volta hardware – due to limited hardware support on older Volta hardware. Nodes within the V100 cluster consist of 8-GPU setups connected with first-generation NVLink in a Hybrid Cube Mesh (HCM) topology. We compute the runtime of communication and computa...

    2024

  9. [1988]

    Efficient parallelization layouts for large-scale distributed model training.arXiv preprint arXiv:2311.05610,

    Johannes Hagemann, Samuel Weinbach, Konstantin Dobler, Maximilian Schall, and Gerard de Melo. Efficient parallelization layouts for large-scale distributed model training.arXiv preprint arXiv:2311.05610,

    arXiv

  10. [2000]

    Olmo: Accelerating the science of language models.arXiv preprint arXiv:2402.00838,

    11 Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models.arXiv preprint arXiv:2402.00838,

    arXiv

  11. [2018]

    Zhenkun Cai, Xiao Yan, Kaihao Ma, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu

    URLhttp://github.com/jax-ml/jax. Zhenkun Cai, Xiao Yan, Kaihao Ma, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu. Tensoropt: Exploring the tradeoffs in distributed dnn training with auto-parallelism.IEEE Transactions on Parallel and Distributed Systems, 33(8):1967–1981,

    1967

  12. [2019]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  13. [2020]

    Branch-train-merge: Embarrassingly parallel training of expert language models

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. InFirst Workshop on Interpolation Regularizers and Beyond at NeurIPS 2022,

    2022

  14. [2021]

    Training deep nets with sublinear memory cost.arXiv preprint arXiv:1604.06174,

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost.arXiv preprint arXiv:1604.06174,

    arXiv

  15. [2022]

    Diloco: Distributed low-communication training of language models

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105,

    arXiv

  16. [2023]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  17. [2024]

    Saptadeep Pal, Eiman Ebrahimi, Arslan Zulfiqar, Yaosheng Fu, Victor Zhang, Szymon Migacz, David Nellans, and Puneet Gupta

    [Accessed 30-09-2024]. Saptadeep Pal, Eiman Ebrahimi, Arslan Zulfiqar, Yaosheng Fu, Victor Zhang, Szymon Migacz, David Nellans, and Puneet Gupta. Optimizing multi-gpu parallelization strategies for deep learning training.Ieee Micro, 39(5):91–101,

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.