{"id":"575e93ad-fe93-4668-b930-84b27a5d4d71","arxiv_id":"2501.04266","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A topology-aware three-level partitioning of weights, gradients, and optimizer states speeds up ZeRO++ LLM training on Frontier by roughly 1.7x per GPU for 20B models with 0.94 scaling efficiency.","lead":"Researchers tuned a memory-saving distributed training method, ZeRO++, for AMD-based Frontier supercomputers by splitting model weights, gradients, and optimizer states across different hardware layers to reduce slow network traffic. In tests with a 20-billion-parameter GPT model on up to 384 GPUs, the tuned version ran about 1.7 times faster per GPU than ZeRO++.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Communication-volume model omits the cross-node gradient Allreduce that §V.C requires; this unaccounted inter-node cost scales with node count and could invalidate the 0.94 scaling-efficiency and 1.71x speedup claims.","rationale":"I agree with the reader's weakest-assumption identification. The single most load-bearing flaw is the omitted cross-node gradient Allreduce introduced in §V.C. This is not a disagreement about community consensus or a stylistic concern; it is an internal accounting gap: the design text itself introduces the Allreduce, yet Tables VII and VIII never list its volume or latency. The manuscript's other limitations—single platform, single model family, no error bars—are acknowledged external-scope issues; the Allreduce omission directly undermines the central quantitative claim. The paper does earn credit for porting ZeRO++ to AMD GPUs and for a topology-aware design, but the burden is on the authors to show that the missing collective is cheap or fully overlapped. A conditional accept is therefore the right posture: the authors should complete the communication accounting, provide profiler data showing the Allreduce's contribution, and rerun the scaling-efficiency analysis. The reader's CONDITIONAL verdict should stand unchanged.","tokens_in":13923,"tokens_out":6590,"duration_ms":62816,"concrete_test":"Extend the volume model in §V.D to include the cross-node gradient Allreduce for GPT-NeoX-20B at 384 GCDs (N=48, P=8). Assume a ring Allreduce on each node-local shard (size ψ/8 per GCD), giving per-node inter-node volume ≈ 2(N−1)/N × ψ ≈ 2ψ. Recompute the predicted per-step communication time using Frontier's 100 GB/s inter-node bandwidth, and compare with ZeRO++'s quantized gradient Reduce-scatter volume (ψ/4). If the added term changes the predicted TFLOPS/GPU gap by more than 10%, the stated scaling-efficiency result must be revised. Independently, instrument the implementation with an RCCL profiler at 384 GCDs and report the fraction of step time spent in this Allreduce.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the communication-volume analysis in §V.D (Tables VII and VIII). Table VIII accounts only for the intra-node gradient Reduce-scatter (volume ψ/4 × (d−1)/d with d=P). However, §V.C explicitly requires a cross-node Allreduce after the intra-node Reduce-scatter: since model weights are replicated across every MI250X but optimizer states are sharded across all GCDs, node-local gradient shards must be averaged across all nodes before parameter updates. This Allreduce moves roughly the full gradient size per node, and its volume grows with node count, so it cannot be folded into the constant-size intra-node Reduce-scatter alone. The paper neither includes this volume in any table nor provides an overlap or latency-hiding argument. Without this term, the claimed per-step inter-node communication reduction over ZeRO++ is incomplete. If the Allreduce is not fully hidden, the inter-node traffic of the proposed design becomes comparable to ZeRO++'s gradient communication, which would erode both the 1.71x TFLOPS/GPU speedup and the 0.94 scaling efficiency at 384 GCDs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a topology-aware extension of ZeRO++ for LLM training on the Frontier supercomputer, based on a three-level hierarchical partitioning of weights (across two GCDs), gradients (across eight GCDs within a node), and optimizer states (across all GCDs), combined with block-based quantization for communication and secondary weight partitions. The authors report a 1.71x increase in TFLOPS per GPU over ZeRO++ and a scaling efficiency of 0.94 for a 20B GPT model at up to 384 GCDs, as well as convergence loss curves matching ZeRO-3. The paper also describes a porting of ZeRO++ to AMD GPUs and a hardware topology analysis of the DGX-A100 versus Frontier nodes.","tokens_in":14158,"tokens_out":3958,"duration_ms":37354,"significance":"If the measured performance holds, the paper is a useful systems contribution: it appears to be the first adaptation of ZeRO++ to AMD GPUs, it exploits Frontier's three-level bandwidth hierarchy, and it applies quantization to reduce-scatter as well as allgather collectives. The inclusion of convergence curves and scaling studies strengthens the empirical case. However, the central communication-cost argument is incomplete, because the cross-node Allreduce required by the design is absent from the volume analysis; this gap directly affects the scaling-efficiency claim. The paper would be suitable for a systems venue after the missing communication term is accounted for and the evaluation methodology is clarified.","major_comments":[{"comment":"The communication-volume analysis in §V.D omits the cross-node Allreduce that the design requires. Section V.C states that after the intra-node Reduce-scatter, \"we call Allreduce on local gradients stored among nodes\" to synchronize gradient shards before weight updates, because optimizer states are sharded across all GCDs while weights are replicated only across pairs of GCDs. This Allreduce moves inter-node traffic that scales with node count and with the gradient shard size; it is not included in Table VIII, nor is any overlap or latency-hiding argument provided. The paper must either add this term to the volume model or show empirically that it is fully hidden; otherwise the claimed inter-node communication reduction and the 0.94 scaling efficiency in Figure 7b do not follow from the stated analysis.","section":"V.D, Tables VII-VIII; V.C"},{"comment":"The scaling-efficiency metric is never defined. The abstract and Figure 7b report a scaling efficiency of 0.94 at up to 384 GCDs, but no baseline configuration is specified (e.g., efficiency relative to 8, 24, or 48 GCDs), and no raw TFLOPS/GPU numbers or repeated-run variance are given for any configuration. Please state the formula and baseline, and report per-configuration measurements with at least the run-to-run spread, since the central speedup and efficiency claims rest on these points.","section":"VI, Figure 7"}],"minor_comments":[{"comment":"The text says each GCD hosts 1.5ψ bytes of weight memory, but with Sec-Degree=8 the table gives 2ψ/2 + ψ/8 = 1.125ψ bytes; 1.5ψ corresponds to Sec-Degree=2. Please reconcile the text with the table.","section":"V.A, Table V"},{"comment":"There are typographical artifacts \"ψψψ\" in several formulas (e.g., \"2ψψψ/8\" and \"ψψψ/2\") that should read \"2ψ/8\" and \"ψ/2\".","section":"Tables V and VIII"},{"comment":"References [13] and [27] are the same paper (PyTorch FSDP) and should be merged or differentiated.","section":"References"},{"comment":"The paper lists MFU as future work, but adding MFU would help substantiate the efficiency claims beyond TFLOPS per GPU.","section":"VII.C"},{"comment":"The y-axes and legends lack units; please specify whether TFLOPS per GPU is measured, theoretical-peak-normalized, and how samples per second is defined.","section":"Figures 7 and 8"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious systems paper from the OSU group, porting ZeRO++ to AMD MI250X and adding a three-level hierarchical partitioning tuned to Frontier's topology. The measured 1.71x TFLOPS/GPU over ZeRO++ on a 20B model at 384 GCDs is the headline, and the convergence curves look fine. But the paper's own communication-volume model has a hole, and the empirical evidence is thinner than it should be.\n\nWhat's new: it's the first ZeRO++ adaptation to AMD GPUs (to my knowledge), and the topology-aware choice of shard degrees — weights across 2 GCDs, gradients across 8 GCDs within a node, optimizer states across all workers — is a reasonable engineering instantiation of ideas from ZeRO++, MiCS, and AMSP. The hardware analysis comparing DGX-A100 and Frontier is clear and useful. The loss curves for 10B and 20B show no obvious convergence regression from quantization.\n\nThe soft spots: the communication analysis in Tables VII-VIII omits the cross-node Allreduce that the design explicitly requires in Section V.C to average gradient shards across nodes. That is not a nitpick — the volume of that Allreduce is on the order of psi/2 per rank, and its latency scales with node count, so it belongs in the table if the paper is arguing for inter-node traffic reduction over ZeRO++. The paper also mentions an Allgather of updated weights after the optimizer step, with volume psi, which is easy to miss in the total. Adding those two gives an inter-node volume that looks comparable to or larger than ZeRO++'s, so the claimed 1.71x speedup is not backed by the paper's own model; it may come from overlap or from Frontier's bandwidth ratios, but the paper doesn't say. There are also no raw throughput numbers, no error bars or repeated runs, and the scaling-efficiency baseline is not defined. The paper doesn't ship code or configs, so the measurements are hard to verify.\n\nThis paper is for the HPC/systems community working on large-scale distributed training, especially on AMD platforms. I would not cite the speedup number until the analysis is fixed, but the topology analysis and the porting effort have standalone value.\n\nRecommendation: worth a serious referee, because the problem is important and the experimental setup is real — but the revision needs a complete communication accounting, a defined scaling baseline, and released artifacts. As is, the central claims are plausible but not fully supported.","headline":"A competent Frontier-specific engineering extension to ZeRO++ with plausible but under-evidenced speedups; the communication-volume accounting is incomplete and needs a second pass.","tokens_in":14730,"tokens_out":10670,"would_cite":false,"duration_ms":101605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-level sharding scheme confines weight and gradient traffic to the fastest links on Frontier, yielding a 1.71x throughput gain over ZeRO++ at 384 GCDs.","keywords":["LLM training","ZeRO++","hierarchical partitioning","communication optimization","AMD MI250X","Frontier supercomputer","gradient quantization","low-bandwidth partitioning"],"falsifier":"Instrument the cross-node gradient Allreduce separately during a 384-GCD 20B run and measure its time as a function of node count; if its cost scales with the number of nodes rather than staying near zero or hidden behind compute, the volume table understates communication and the scaling claim fails.","tokens_in":13728,"feed_emoji":"⚡","tokens_out":6742,"duration_ms":63008,"temperature":0.7,"pith_summary":"Large language model training with ZeRO-3 shards weights, gradients, and optimizer states across all workers, but every training step then pays for expensive node-crossing Allgather and Reduce-scatter operations. This paper claims that on Frontier, those collectives can be deliberately confined to the fast layers: model weights live on the two compute dies inside one MI250X GPU, gradients live on the eight dies inside one node, and only optimizer states are sharded across all nodes. The measured result is a 1.71x increase in TFLOPS per GPU over ZeRO++ for a 20B-parameter GPT model at 384 GCDs, with scaling efficiency of 0.94. If the claim holds, it shows that topology-aware sharding plus block quantization can turn a low-bandwidth supercomputer interconnect into a near-linear scaling platform for billion-parameter training.","feed_headline":"Three-level sharding lifts Frontier LLM training 1.71x over ZeRO++","feed_subtitle":"Confining weight and gradient traffic to fast on-chip links keeps scaling efficiency at 0.94 up to 384 GCDs.","key_machinery":"The central object is a three-level hierarchical partition indexed by Frontier's three bandwidth layers: GCD-to-GCD inside an MI250X (fastest), GPU-to-GPU inside a node via Infinity Fabric, and inter-node Slingshot (slowest). The scheme fixes the weight shard factor at 2, the gradient shard factor at 8 (one node), and the optimizer-state shard factor at the total number of GCDs. These choices satisfy the dependency rule that each worker only holds gradients and optimizer states for parameters it owns, and they make the device count for weight and gradient collectives independent of job size. Communication volume is further cut with block-wise quantization: INT8 for weight Allgather and secondary partitions, INT4 for gradient Reduce-scatter.","core_discovery":"The central claim is that ZeRO-3's communication bottleneck on Frontier is avoidable by assigning each model state a sharding factor matched to a bandwidth tier. The proposed scheme shards FP16 model weights across the two GCDs inside a single MI250X package (shard degree 2), shards gradients across the eight GCDs inside one compute node (shard degree 8), and shards optimizer states across all GCDs in the job, mirroring ZeRO-3. Weight Allgather for forward and backward passes then runs only over the fast GCD-to-GCD links inside one MI250X, gradient Reduce-scatter runs only over intra-node links, and INT8/INT4 block quantization halves or quarters the bytes moved. The paper reports up to 139.8% higher TFLOPS per GPU than ZeRO-3 and 70.7% higher than ZeRO++ for a 20B model at 384 GCDs, a 1.71x gain over ZeRO++, and scaling efficiency of 0.94, with loss curves close to unquantized ZeRO-3.","pith_inferences":["The paper only validates on Frontier with AMD MI250X; the same shard-degree recipe would need recalibration on systems with different bandwidth ratios, and the general lesson is the ratio, not the absolute numbers.","The volume analysis omits the cross-node Allreduce that synchronizes gradient shards across nodes; whether that call stays hidden behind computation is the main untested scaling risk beyond 384 GCDs.","Because per-device weight and gradient memory is fixed, comparing this scheme with ZeRO-3 involves a crossover: at small node counts ZeRO-3 uses less per-device memory, while at large node counts this scheme avoids ZeRO-3's growing inter-node traffic."],"forward_implications":["At 384 GCDs a 20B-parameter model trains at 1.71x the TFLOPS per GPU of ZeRO++, and scaling efficiency reaches 0.94, so jobs that previously slowed at large node counts can run near-linearly.","Weight Allgather latency becomes constant as nodes are added because it touches only two GCDs, and gradient Reduce-scatter latency becomes constant because it stays inside a node.","Block quantization keeps the training loss close to unquantized ZeRO-3 while reducing bytes moved, so the throughput gain does not come at an obvious convergence cost in the measured runs.","The scheme trades memory for bandwidth: weights and gradients occupy fixed per-device memory (1.5ψ and ψ/4 bytes), which lets large models run on small node counts but caps practical model size near 36B parameters."],"supporting_citations":[{"why":"Defines ZeRO-3 sharding and the Allgather/Reduce-scatter patterns this paper re-partitions.","marker":"[12]"},{"why":"ZeRO++ is the baseline and provides the quantization-assisted Allgather/Reduce-scatter kernels and secondary weight partitions this work extends.","marker":"[18]"},{"why":"Supplies the sharding dependency rule and the idea of independent partition sizes for different model states.","marker":"[11]"},{"why":"Block-wise quantization is the method used to compress weights and gradients with small accuracy loss.","marker":"[22]"},{"why":"Documents Frontier's node and MI250X topology, the bandwidth tiers the design is built around.","marker":"[16]"},{"why":"Describes the Slingshot inter-node interconnect whose low bandwidth motivates avoiding cross-node collectives.","marker":"[17]"},{"why":"GPT-NeoX-20B is the model used in the scaling and loss-curve evaluations.","marker":"[24]"}],"fun_headline_variants":["Tiered sharding matches link speeds: 1.71x LLM training gain on Frontier","Low-bandwidth sharding lifts Frontier LLM training 1.71x over ZeRO++","Sharding by bandwidth tier yields 1.71x faster LLM training on Frontier","Frontier LLM training: hierarchical sharding gives 1.71x speedup at 384 GPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis counts only the intra-node Reduce-scatter volume for gradients and silently assumes the cross-node Allreduce that synchronizes gradient shards across nodes is cheap or fully overlapped; if that Allreduce grows with node count, the claimed communication savings and the 0.94 scaling efficiency do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Tiered sharding matches link speeds: 1.71x LLM training gain on Frontier","Low-bandwidth sharding lifts Frontier LLM training 1.71x over ZeRO++","Sharding by bandwidth tier yields 1.71x faster LLM training on Frontier","Frontier LLM training: hierarchical sharding gives 1.71x speedup at 384 GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2960,"prompt_tokens":1013,"completion_tokens":1947,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1846}},"tokens_in":629,"tokens_out":1947,"duration_ms":16446,"temperature":1.0,"reasoning_tokens":1846,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:37:44.879561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the cross-node gradient Allreduce separately during a 384-GCD 20B run and measure its time as a function of node count; if its cost scales with the number of nodes rather than staying near zero or hidden behind compute, the volume table understates communication and the scaling claim fails.","supporting_citations":[{"cited_title":"AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training","cited_arxiv_id":"2311.00257","evidence_quote":"Supplies the sharding dependency rule and the idea of independent partition sizes for different model states."},{"cited_title":"Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 | TOP500 — top500.org,","cited_arxiv_id":null,"evidence_quote":"Documents Frontier's node and MI250X topology, the bandwidth tiers the design is built around."},{"cited_title":"An in-depth analysis of the slingshot interconnect,","cited_arxiv_id":null,"evidence_quote":"Describes the Slingshot inter-node interconnect whose low bandwidth motivates avoiding cross-node collectives."},{"cited_title":"GPT-NeoX-20B: An open-source autoregressive language model,","cited_arxiv_id":null,"evidence_quote":"GPT-NeoX-20B is the model used in the scaling and loss-curve evaluations."}],"review_version":1}