{"id":"bb55c03a-d8a0-450c-ac1c-11792e1d1e29","arxiv_id":"2411.13055","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"At 128 or more GPUs, FSDP training becomes communication-bound, making tensor and pipeline parallelism preferable, and extra GPUs yield diminishing throughput per watt.","lead":"This paper shows how training large language models across thousands of GPUs loses efficiency as communication overhead grows. It finds that splitting the model across GPUs can outperform data-only sharding at scale, and that adding GPUs gives shrinking returns in tokens per watt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'properly optimized' claim depends on an untested parallelization family: HSDP shrinks the very FSDP AllGather/ReduceScatter world size that drives the paper's main trend, so the reported diminishing returns could flatten under a hierarchical sharding strategy.","rationale":"The paper is a well-executed empirical study: the NCCL primitive measurements in Figure 2, the kernel-level profiling in Figures 3-5, and the parallelization sweeps in Figures 6-8 are internally consistent and support the claim that flat FSDP becomes communication-bound at scale. The reader's CONDITIONAL verdict is appropriate. The weakest point is external validity: the headline statement that returns diminish 'even when hardware and parallelization strategies are properly optimized' is only established over the strategies actually swept. HSDP is the most direct challenge because it attacks exactly the AllGather/ReduceScatter cost that generates the paper's main effect, and the authors themselves list it as future work in Appendix A. This is a missing-support problem rather than an internal inconsistency, so it should change the confidence placed in the general recommendation without overturning the measured trends. Adding one comparative HSDP experiment would settle whether the central claim is a property of large-scale distributed training or of the particular FSDP sharding layout studied here.","tokens_in":17447,"tokens_out":4999,"duration_ms":54964,"concrete_test":"Run Llama-7B with per-device local batch size 2 on 256 and 2048 H100 GPUs using PyTorch HSDP (sharding group per 8-GPU node, replica group across nodes) with the same prefetching and no parameter resharding settings as §3, and compare WPS, MFU, and tokens/Watt against the best tensor/pipeline configuration reported in Figures 6 and 10b. If HSDP matches or exceeds the best model-parallel throughput at 2048 GPUs, the 'properly optimized' claim and the model-parallelism recommendation need revision; if HSDP is worse, the paper's central scaling conclusion is robust to this omission.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims diminishing returns 'even when hardware and parallelization strategies are properly optimized,' but the optimization search in §4.3 is restricted to tensor and pipeline parallelism combined with flat FSDP sharding. Appendix A explicitly defers Hybrid Sharded Data Parallelism (HSDP) to future work. This is not a marginal omission: HSDP shards parameters and gradients only within a node and uses AllReduce across nodes, reducing the NCCL AllGather/ReduceScatter world size that §2.2 and Figure 2 identify as the scaling bottleneck, without adding tensor-parallel AllReduces or pipeline bubbles. The paper's central observation that FSDP becomes communication-bound beyond roughly 128 GPUs and that model parallelism recovers throughput (Figures 3, 6, 10b) is therefore conditional on a flat sharding group. If HSDP at 2048 H100 GPUs keeps the expensive collectives inside 8-GPU nodes, the throughput-per-added-GPU curve could degrade much more slowly, and the reported +52.60% improvement from tensor parallelism may not represent a properly optimized configuration. The measurements themselves remain valid evidence about flat FSDP, but the general 'even when properly optimized' conclusion overreaches past the tested strategy space.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an extensive empirical study of distributed training of Llama-2-style LLMs (1B to 70B parameters) on clusters ranging from 8 to 2048 H100/A100 GPUs, using PyTorch FSDP combined with optional tensor and pipeline parallelism. The central measurements are global and per-device throughput, exposed communication (from profiled NCCL/CUDA kernel times), MFU, and GPU power draw. The paper reports three main findings: (1) FSDP's AllGather/ReduceScatter communication overhead grows with data-parallel world size, making model parallelism (tensor or pipeline, at degrees 2--4) beneficial at scales beyond roughly 128 GPUs; (2) global throughput increases sublinearly with added accelerators while per-GPU power draw remains nearly flat, so tokens-per-watt and MFU decline with scale; and (3) the optimal configurations for sharded training differ from earlier recommendations that model parallelism be avoided. The abstract and conclusion generalize these results to claim that scaling accelerators yields diminishing returns 'even when hardware and parallelization strategies are properly optimized.'","tokens_in":17682,"tokens_out":9866,"duration_ms":89824,"significance":"If the conclusions hold, they provide actionable guidance for training large models at scale: flat FSDP becomes communication-bound above a few hundred GPUs, and moderate tensor/pipeline parallelism is preferable; they also highlight an energy-efficiency penalty of scale that is largely absent from existing scaling-law discussions. The paper's strengths are the unusual scale of the measurements (up to 2048 H100s), the breadth across model sizes and hardware generations, the detailed kernel-level accounting of exposed communication, and the fact that the trends are direct observations rather than outputs of a fitted parametric model. The central caveat is that the 'properly optimized' claim is conditional on a limited strategy family; this affects the generality of the diminishing-returns conclusion, not the validity of the measurements themselves.","major_comments":[{"comment":"The abstract and Section 5 assert diminishing returns 'even when hardware and parallelization strategies are properly optimized,' but the optimization search in Section 4.3 is restricted to tensor and pipeline parallelism of degree at most 16 combined with a flat FSDP sharding group. Appendix A explicitly defers Hybrid Sharded Data Parallelism (HSDP) and other hierarchical sharding strategies to future work. This is not a peripheral omission: Sections 2.2 and Figure 2 identify AllGather/ReduceScatter world size as the scaling bottleneck, and HSDP is precisely a way to keep those collectives inside 8-GPU nodes. If HSDP (or a similar hierarchical scheme) performs sufficiently better at 2048 GPUs, the reported diminishing-returns curve and the +52.60% improvement from tensor parallelism may not represent the 'properly optimized' configuration. The measurements remain valid as a characterization of flat FSDP, but the general claim should be scoped to the considered parallelism family, or HSDP experiments should be added at the largest scales.","section":"Abstract, §4.3, Appendix A"},{"comment":"Section 3 states that metrics are aggregated over 50 training iterations after 10 warmup iterations, but this appears to be a single run and no run-to-run variance or confidence intervals are reported. Because the central quantitative claims (e.g., the 37.22% per-device throughput decrease from 128 to 2048 GPUs in Section 4.1 and the +52.60% WPS improvement from tensor parallelism in Section 5) are used to support the diminishing-returns conclusion, the absence of variance information makes it impossible to judge whether the differences between configurations are statistically significant. Please report standard deviations over at least a few repeated runs (or justify why the environment is deterministic and demonstrate this with measurements).","section":"§3 (Metrics), §4.1, §4.3"},{"comment":"Section 4.2 and Figure 5 describe strong scaling with a 'fixed global batch size of 32 examples' across 2 to 32 nodes, but with 256 GPUs and global batch 32, a pure data-parallel configuration would require a sub-unit per-device batch. Presumably the optimal configurations use enough model parallelism to keep the data-parallel degree no larger than 32, but the text does not state how the configurations were re-searched per node count. The caption also says the optimal strategy is 'determined by the experimental results displayed in Figure 6,' which is for a different setup (256 GPUs, global batch 512). Please clarify the selection procedure and the feasibility of the global batch at each scale; this is load-bearing for the strong-scaling results.","section":"§4.2, Figure 5"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'langauge' in the introduction, 'Zero-3' instead of 'ZeRO-3' in Section 6, and inconsistent capitalization of 'FLOPS'/'FLOPs' throughout.","section":"Introduction, Related Work"},{"comment":"The units for AllReduce/AllGather bandwidth in Figure 2 are reported as GB/s, while Appendix B refers to per-node InfiniBand bandwidth in GB/s; please clarify whether these are GB/s or Gb/s, and whether the 'favorable scaling' of AllReduce bandwidth is specific to the tree algorithm or a general statement.","section":"§2.2, Figure 2, Appendix B"},{"comment":"The text says 'nearly 20% decrease in hardware utilization' when comparing A100 to H100 (59.67% to 40.77%). This is a decrease of about 19 percentage points but roughly 32% relative; please report the relative and absolute changes explicitly to avoid ambiguity.","section":"§4.4"},{"comment":"Power efficiency is based on NVML GPU power draw only; the paper should state explicitly that total system power (including CPU, memory, and network) is not included, so the energy-efficiency claims apply to GPU power consumption.","section":"§3 (Power utilization)"},{"comment":"In Figure 13, the y-axis for 'Exposed Communication' is labeled 'Time (microseconds)' but the values are on the order of 1e6, which is inconsistent; the unit is likely milliseconds. Please fix the axis label.","section":"Appendix F, Figure 13"}],"recommendation":"major_revision","confidential_remarks":"The main reason for major revision is the gap between the paper's broad 'properly optimized' conclusion and the limited search space, a gap the authors themselves acknowledge in Appendix A. This is not a hidden flaw, but it is load-bearing for the central claim and should be addressed either by adding HSDP experiments or by scoping the claims. The measurements themselves are large-scale, clearly described, and likely to be of wide interest; with appropriate reframing or additional experiments, the paper is publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a badly needed measurement paper, and the main qualitative finding — FSDP's AllGather/ReduceScatter costs grow with world size and model parallelism recovers throughput — is well supported by the data up to 2048 H100s. It deserves serious refereeing. But the abstract's 'even when hardware and parallelization strategies are properly optimized' overreaches: the tested search is flat FSDP with tensor/pipeline parallelism up to degree 16, and Hybrid Sharded Data Parallelism (HSDP) is explicitly left to future work. HSDP is not a minor variant. It shrinks the exact AllGather/ReduceScatter world size that the paper identifies as the bottleneck, replacing cross-node sharding with an AllReduce that scales far better. The diminishing-returns curve is therefore a statement about flat FSDP plus TP/PP, not about all reasonable sharding strategies. That scope limitation should be in the abstract and conclusion, not just the appendix.\n\nWhat's genuinely new: prior empirical scaling studies (Narayanan et al., Hagemann et al.) measured 3D parallelism without FSDP and concluded model parallelism should be minimized. This paper supplies measurements in the FSDP/ZeRO regime at scales those studies did not reach, and shows the opposite preference. The power-efficiency angle — total power draw rises roughly linearly while per-device throughput falls — is a useful addition, and the NCCL primitive benchmarks in Figure 2 give a mechanistic explanation for the trend.\n\nSoft spots, in order: (1) The 'properly optimized' completeness issue above. (2) No reported run-to-run variance. The figures aggregate 50 iterations, but there are no repeated runs, so we can't tell how stable the +52.60% tensor-parallel gain is. The trend is large enough that I doubt it flips, but a systems paper with this much quantitative weight should give error bars or at least confidence intervals on the headline numbers. (3) No code, configs, or traces released. That limits reproducibility, though Meta cluster constraints may make this hard.\n\nWhat holds up: the central measured trends are internally consistent across Figures 3, 5, 6, 7, and 10, and consistent with NCCL ring AllGather behavior. The limitation appendix is honest about HSDP, activation checkpointing, and async methods. So the paper is not covertly overclaiming; the overclaim is in the headline wording, not in the body.\n\nWho it's for: anyone choosing parallelization for large LLM training runs, and anyone who uses FLOP-based scaling laws that ignore communication. I'd send it to referees and ask them to push on the scope of the 'properly optimized' claim and on variance/reproducibility. The HSDP concern should be raised to the authors as a required discussion point, not treated as fatal.","headline":"Solid, wide-ranging FSDP scaling study with a real finding; the headline claim of 'properly optimized' overreaches past the untested HSDP family.","tokens_in":18220,"tokens_out":3066,"would_cite":true,"duration_ms":31692,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Beyond roughly 128 GPUs, model parallelism outperforms data-parallel sharding for LLM training.","keywords":["distributed training","Fully Sharded Data Parallelism","model parallelism","communication overhead","scaling laws","power efficiency","LLM training","throughput"],"falsifier":"Train Llama-7B at 2048 H100 GPUs with hybrid sharded data parallelism (sharding only within a node) or with ZeRO-3 using node-local sharding, under the same fixed global batch size and measured with the paper's methodology, and compare per-GPU throughput and MFU to the paper's best tensor-parallel configuration. If per-GPU throughput fails to decline by roughly a third relative to 128 GPUs, or MFU stays above 40%, the claimed diminishing returns under proper optimization would not hold.","tokens_in":17239,"feed_emoji":"📉","tokens_out":5138,"duration_ms":45075,"temperature":0.7,"pith_summary":"This paper tries to establish that large-scale distributed training of language models has two under-appreciated properties: past a scale of roughly 128 GPUs, the communication overhead of fully sharded data parallelism (FSDP) is large enough that adding tensor or pipeline parallelism—traditionally viewed as a utilization tax—actually increases throughput; and beyond a few hundred GPUs, additional accelerators deliver shrinking per-GPU returns even when the parallelization strategy is chosen well, because per-GPU power draw stays roughly constant while per-GPU throughput falls. The authors support this with measurements across 1B to 70B parameter models on up to 2048 H100 and A100 GPUs. If correct, the results imply that the common practice of sharding parameters and scaling out data-parallel replicas is compute- and energy-inefficient at pretraining scale, and that scaling laws built on FLOP counts alone will overestimate real throughput.","feed_headline":"Model parallelism wins past 128 GPUs in LLM training","feed_subtitle":"A 2048-GPU study finds throughput per chip falls while power draw stays flat, even with optimized setups.","key_machinery":"The load-bearing object is the exposed-communication mechanism of Fully Sharded Data Parallelism: parameter materialization and gradient reduction go through NCCL AllGather and ReduceScatter collectives, which use ring algorithms whose latency scales with node count (measured in Figure 2), unlike the tree-based AllReduce used by standard data parallelism and tensor parallelism. Because communication-bound kernels cannot be hidden by computation once the per-device compute time is small relative to collective latency, exposed communication grows and hardware utilization collapses. Model parallelism is the lever that shrinks the FSDP collective's world size, trading AllReduce and pipeline-bubble costs for reduced AllGather/ReduceScatter volume, which the paper measures as lower exposed communication and higher global throughput.","core_discovery":"The central discovery is that communication boundedness, not compute, governs large-scale FSDP training. The AllGather and ReduceScatter collectives used to rematerialize and update sharded parameters are ring-based and their latency grows with world size, so as nodes are added the ratio of communication to computation rises; beyond 128 H100 GPUs for a 7B model, exposed communication is unavoidable, hardware utilization (MFU) falls, and throughput per device drops by 37.22% when scaling from 128 to 2048 GPUs. Because GPU power draw declines only slightly (5.87%) while throughput falls, tokens per watt decline markedly. The paper shows that moderate model parallelism (tensor or pipeline degrees 2–4) shrinks the data-parallel group sizes for FSDP collectives and thereby reduces exposed communication, improving throughput, MFU, and power efficiency relative to the data-parallel baseline—including a +52.60% words-per-second gain at 2048 GPUs. It also shows that faster compute relative to interconnect (H100 vs A100) worsens communication boundedness, and that scaling model size increases both compute and communication jointly.","pith_inferences":["The untested strategies the paper defers, especially hybrid sharded data parallelism that keeps AllGather inside a node, could plausibly shift the crossover point where model parallelism becomes preferable; the reported curves may be an upper bound on what current FSDP-like training can achieve.","The diminishing-returns pattern likely extends to TPU-based training and to inference-time parallel decoding, because the same ring-collective latency growth applies whenever sharded parameters must be gathered across many devices.","A testable design implication is that node-local sharding with inter-node only AllReduce—the hybrid pattern—should recover much of the lost throughput, which would directly quantify how much of the reported 37% drop is due to the ring AllGather world size.","Scaling-law papers that ignore communication will increasingly mispredict time-to-train as clusters grow; replacing FLOP count with a communication-aware cost measure is a tractable near-term correction."],"forward_implications":["Practitioners training 7B-class models on clusters above roughly 128 GPUs should add tensor parallelism at degree 2 or 4 instead of relying on FSDP data-parallel replicas, because it reduces exposed communication and raises global throughput.","Throughput-based scaling laws that assume linear gains with added accelerators will overestimate output at pretraining scale; compute-optimal scaling should incorporate communication cost.","Energy per token rises with cluster size, so the environmental cost of a training run is not proportional to FLOPs but to a less favorable curve.","Faster GPU compute alone will not fix the problem; interconnect and memory bandwidth improvements are needed to prevent training from becoming more communication-bound."],"supporting_citations":[{"why":"Defines Fully Sharded Data Parallelism, the sharded-data-parallel baseline whose collectives the paper measures.","marker":"Zhao et al. (2023)"},{"why":"Introduces DeepSpeed ZeRO, the sharded-data-parallel family that motivates the FSDP communication analysis.","marker":"Rasley et al. (2020)"},{"why":"Provides Megatron-LM tensor/model parallelism, the 3D-parallel framework the paper combines with FSDP.","marker":"Shoeybi et al. (2019)"},{"why":"The prior 3D-parallelism study whose recommendation to minimize model parallelism the paper overturns.","marker":"Narayanan et al. (2021)"},{"why":"The prior parallelization-layout study the paper extends and contradicts regarding the value of model parallelism in sharded settings.","marker":"Hagemann et al. (2023)"}],"fun_headline_variants":["Scaling LLM training past 128 GPUs yields diminishing returns","Communication bottleneck caps LLM training scale at 128 GPUs","Model parallelism boosts LLM training efficiency beyond 128 GPUs","Throughput per GPU drops 37% when scaling LLM training to 2048"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that returns diminish 'even when hardware and parallelization strategies are properly optimized' assumes the tested sweep over tensor and pipeline parallelism (degrees up to 16 on fixed node counts) covers the strategies a truly optimized system would use; the paper does not test hybrid sharded data parallelism, ZeRO-3 with different sharding policies, or newer fabric topologies, any of which could in principle flatten the diminishing-returns curve.","fun_headline_variants_meta":{"raw":{"variants":["Scaling LLM training past 128 GPUs yields diminishing returns","Communication bottleneck caps LLM training scale at 128 GPUs","Model parallelism boosts LLM training efficiency beyond 128 GPUs","Throughput per GPU drops 37% when scaling LLM training to 2048"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1385,"prompt_tokens":983,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":599,"tokens_out":402,"duration_ms":4302,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:52:51.255579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Llama-7B at 2048 H100 GPUs with hybrid sharded data parallelism (sharding only within a node) or with ZeRO-3 using node-local sharding, under the same fixed global batch size and measured with the paper's methodology, and compare per-GPU throughput and MFU to the paper's best tensor-parallel configuration. If per-GPU throughput fails to decline by roughly a third relative to 128 GPUs, or MFU stays above 40%, the claimed diminishing returns under proper optimization would not hold.","supporting_citations":[],"review_version":1}