{"id":"56c8da6b-ce17-45ee-8a6d-b585cd0ea694","arxiv_id":"2508.13337","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"X-MoE combines padding-free token buffers, redundancy-bypassing dispatch, and sequence-sharded MoE blocks to train a 545B-parameter DeepSeek-style MoE on 1024 AMD MI250X GPUs.","lead":"X-MoE is a training system that runs DeepSeek-style mixture-of-experts models on AMD-powered supercomputers, claiming to train a 545-billion-parameter model on 1024 GPUs. A generalist might read it to see how system engineering can unlock large-scale AI training on non-NVIDIA hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 545B 'trainable' claim rests on a single throughput bar; no loss curve, iteration count, or wall-clock trace is shown, and Appendix D reports severe all-to-all outliers at 1024 GPUs.","rationale":"The paper has real engineering contributions: PFT eliminates padding, RBD quantifiably cuts inter-node traffic, SSMB targets a genuine activation bottleneck, and small-model loss tracking against DeepSpeed is useful evidence. The weakness is not mechanism-level but claim-level: the headline is an end-to-end trainability and throughput statement at 545B, and that specific evidence is missing. Appendix D's own admission of network outliers makes the throughput number vulnerable to measurement-window effects. I agree with the reader's weakest assumption: the central claim should be treated as conditional until a demonstrated 545B training run is provided. The conditional verdict remains appropriate; the mechanisms are plausible and partially validated at smaller scale, so a stronger sanction is not warranted.","tokens_in":26388,"tokens_out":7982,"duration_ms":84701,"concrete_test":"Obtain (or run) the 545B configuration on 1024 GPUs for at least 100 consecutive iterations and report: (a) the LM-loss curve, (b) mean/median/p95 per-iteration wall-clock time, (c) number and duration of all-to-all outliers, and (d) the largest model each baseline (DeepSpeed-MoE, DeepSpeed-TED, Tutel) can actually train on 1024 GPUs under the same sweep. If the 545B run shows decreasing loss and stable step times with outliers quantified, the claim stands; if only memory fit is shown, downgrade to 'memory-feasible, training not demonstrated.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support 'enables training of 545B models' (abstract, §5.2), the paper must show a sustained, numerically valid run, not merely memory feasibility. The only 545B evidence is Fig. 9: one aggregated-throughput number (10.44 PFLOPS) and OOM markers for baselines. No loss curve, gradient statistics, iteration count, or step-time distribution is given for the 545B configuration. §5.6 validates correctness only on the 10.1B model for 500 iterations, and it explicitly identifies a different token-dropping rule (X-MoE drops only on capacity, DeepSpeed also drops negative-score tokens), so small-model equivalence does not automatically transfer to a novel 545B layout. Appendix D is an in-scope limitation: beyond 256 GPUs all-to-all latency shows >10x average outliers and >500 ms collectives at 512/1024 GPUs on the shared cluster. If the 10.44 PFLOPS figure is a favorable-window measurement, the 'high training throughput' and 'sustained training' parts of the claim are unsubstantiated. The '10x larger than the largest trainable model' comparison also lacks a direct 1024-GPU sweep of baseline maximum trainable size; the stated factor is not tied to a measured baseline ceiling. This makes the central claim a memory-footprint result unless a real training trace is supplied.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes X-MoE, a training system for expert-specialized Mixture-of-Experts (MoE) models targeting non-NVIDIA HPC platforms, with evaluation on the AMD MI250X-based Frontier supercomputer. The system combines three main techniques: Padding-Free Token buffers (PFT) with Triton-based sparse kernels that eliminate zero-padding in MoE dispatch/combine, a hierarchical Redundancy-Bypassing Dispatch (RBD) that reduces inter-node all-to-all traffic, and Sequence-Sharded MoE Blocks (SSMB) that reduce activation memory under hybrid parallelism. The evaluation reports that X-MoE can train DeepSeek-style MoEs up to 545B parameters on 1024 GPUs with 10.44 PetaFLOPs aggregated throughput, that it outperforms DeepSpeed-MoE, DeepSpeed-TED, and Tutel in several Frontier configurations, and that it shows mixed but functional portability on 8x A100 GPUs. The source code is made available.","tokens_in":26675,"tokens_out":8955,"duration_ms":87967,"significance":"If the central claims are fully substantiated, this is a strong systems contribution: it directly addresses the activation-memory bottleneck of fine-grained, high-top-k MoEs, provides a portable (Triton-based) implementation on a major non-NVIDIA HPC platform, and releases code that should aid reproducibility. The paper's memory-bottleneck analysis in Section 3.2 and the SSMB-vs-TED trade-off analysis in Appendix C are valuable in themselves. The empirical gains over three established baselines on Frontier are meaningful, and the design is clearly described. The main reservation is that the headline '545B trainable model' claim rests on a throughput bar and memory feasibility rather than on evidence of a sustained, numerically valid training run; this needs to be addressed before the strongest claims can be accepted.","major_comments":[{"comment":"The headline claim that X-MoE 'enables training' of the 545B Super model on 1024 GPUs is not supported by the evidence shown. Figure 9 reports a single aggregated throughput number (10.44 PFLOPS) and OOM markers for the baselines, but no loss curve, gradient-norm trace, iteration count, or step-time distribution is provided for this configuration. Appendix D is an in-scope limitation that directly matters here: it reports that all-to-all collectives at 512 and 1024 GPUs frequently exceed 500 ms and that beyond 256 GPUs latencies are on average over 10x higher due to cross-rack congestion. A throughput measurement over an unspecified window cannot demonstrate sustained, numerically stable training. Please add a training trace (e.g., loss over at least several hundred iterations, with step-time statistics) for the Super configuration, or explicitly narrow the claim to memory feasibility plus a one-step throughput benchmark.","section":"§5.2, Fig. 9, Appendix D"},{"comment":"The '10x larger than the largest trainable model' factor is not tied to a measured baseline ceiling. The paper reports OOM for the baselines at particular configurations in Fig. 9, but it does not sweep baseline configurations on 1024 GPUs to find the largest model each baseline can actually train under the same hardware and memory settings. Baseline memory and throughput depend strongly on EP size and ZeRO stage (which the paper itself sweeps in §5.2), so the 10x factor is not established by the data as presented. Please report, for each baseline, the largest trainable model size at 1024 GPUs under matched parallelism and memory settings, or explicitly state the configuration used to derive the 10x ratio.","section":"§5.2"},{"comment":"The numerical validation covers only the 10.1B model for 500 iterations on 16 GPUs, and the paper itself identifies a difference in token-dropping semantics between X-MoE and DeepSpeed-MoE: X-MoE drops only on capacity, whereas DeepSpeed-MoE also drops tokens with negative routing scores. This is not sufficient to validate the larger configurations, where SSMB, RBD, and higher TP degrees change communication and sharding behavior. In particular, no evidence is shown that the 201B or 545B configurations would produce a stable loss trajectory rather than diverging due to routing or gradient issues at scale. Please provide a correctness check at a multi-node expert-specialized scale (e.g., a loss comparison against a reference implementation, or a gradient-norm check), and either align the token-dropping rules or report explicitly how their difference affects the comparison.","section":"§5.6"}],"minor_comments":[{"comment":"The sentence 'existing systems such as DeepSpeed-MoE, DeepSpeed-TED, and Tutel run out of memory on medium and large models' is contradicted by Fig. 9, where Tutel trains the Medium model at 17.0 TFLOPS and DeepSpeed-MoE attains 4.7 TFLOPS; please revise the wording to describe which systems actually OOM on which model.","section":"§5.2"},{"comment":"The x-axis labels of Fig. 11 call the 55.2B model 'Large', but Table 3 designates it as 'Medium'; please make the naming consistent throughout.","section":"§5.4.1, Fig. 11"},{"comment":"The abbreviations 'Small-SR' and 'Small-LR' are defined in the body text but not in the table caption; please add the definitions to the caption, and state explicitly whether the same EP size, TP degree, and ZeRO stage were used for all three frameworks on the A100 experiments, since these settings directly affect the throughput comparison.","section":"§5.5, Table 5"},{"comment":"The relationship between the 'max_token_count' parameter in Listing 1 and the capacity factor c=1.25 mentioned in §5.1 is not specified; please state the formula (e.g., max_token_count = floor(c * S * K / E)) so that the token-dropping behavior is reproducible.","section":"§4.1.1, Listing 1"},{"comment":"The measurement methodology for the 10.44 PFLOPS figure is not given: please state over how many steps the throughput is averaged and whether all-to-all outlier events are included or excluded, since Appendix D shows that outliers are frequent at 1024 GPUs.","section":"§5.2, Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the 545B claim: the paper's strongest sentence in the abstract goes beyond what the evaluation demonstrates. I would require the sustained-training evidence described in major comment 1 before considering the paper for acceptance. Also, for transparency, the DeepSpeed-MoE baseline paper shares an author with the current paper; this does not by itself make the comparison unfair, but the authors should disclose any code-level reuse or tuning relationship."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a real systems paper, not a packaging exercise. X-MoE builds a padding-free MoE pipeline in Triton, adds redundancy-bypassing dispatch and sequence-sharded MoE blocks, and targets exactly the fine-grained high-top-k MoE regime where activations—not parameters—are the bottleneck. That diagnosis in Section 3.2 is the paper's sharpest contribution, and the layer-level breakdowns back it up: gating/dispatch/combine time drops by 5.7x/35.7x/8.1x on the small model, and alltoall time halves on the large model. The A100 portability experiments are a nice bonus, and the code is on GitHub.\n\nThe soft spot is the headline. \"Enables training up to 545B\" currently rests on Figure 9: one aggregated throughput bar (10.44 PFLOPS), OOM markers for baselines, and no loss curve, iteration count, or step-time distribution for the 545B run. Correctness validation is only on the 10.1B model, 500 iterations, against DeepSpeed-MoE, and the authors themselves note a token-dropping rule difference. Appendix D shows >500ms all-to-all outliers at 512/1024 GPUs, so the 10.44 PFLOPS number could be a favorable-window measurement. Also, the \"10x larger than the largest trainable model\" claim is not tied to a measured 1024-GPU baseline ceiling. These are load-bearing gaps for the scale claim, but they don't invalidate the layer-level engineering, which is measured more carefully.\n\nOne fairness note: DeepSpeed-MoE is a baseline and includes a co-author of this paper. That's not disqualifying, but baseline configurations are not fully reported, so I'd ask for the config details and for either a 545B training trace or a reworded abstract. The comparison looks plausible on the smaller models, where both systems run, but the OOM-only comparisons at scale don't tell us how close the baselines were to their own limits.\n\nBottom line: this deserves a serious referee. The systems contributions are solid, the writeup is honest about the noisy cluster, and the code availability is a plus. I'd send it to peer review with a request for a sustained 545B trace or a softened claim, plus error bars on the Frontier throughput. This is the kind of work that moves MoE training on AMD hardware forward, and the right reviewer will be able to push on the weak point without throwing out the engineering.","headline":"A genuinely useful MoE systems paper with a convincing memory-bottleneck story; treat the 545B 'trainable' claim as a memory-feasibility result until a real training trace is shown.","tokens_in":27271,"tokens_out":2077,"would_cite":true,"duration_ms":21372,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"X-MoE trains 545B-parameter MoEs on 1024 AMD GPUs, a 10x jump over prior systems on the same hardware.","keywords":["mixture-of-experts training","fine-grained experts","large top-k routing","padding-free token buffers","hierarchical all-to-all","sequence sharding","AMD ROCm","HPC training"],"falsifier":"Run the reported 545.4B configuration to a fixed iteration budget with a logged loss curve and compare its slope to the 201B model's; if the run fits in memory but the loss diverges or the run cannot complete iterations, the trainability claim collapses. Counting the number of completed optimizer steps and sustained throughput over those steps would also settle whether the 1024-GPU result is training or a single-step benchmark.","tokens_in":26156,"feed_emoji":"🖥️","tokens_out":6288,"duration_ms":62380,"temperature":0.7,"pith_summary":"X-MoE is a training system for the new generation of mixture-of-experts language models in which many small experts are activated per token (DeepSeek-style, fine-grained experts with large top-$k$ routing). The paper argues that such models hit a memory bottleneck in the dispatch and combine stages, not in expert weights, and that existing MoE frameworks tuned for a few large experts on NVIDIA hardware waste memory and bandwidth on zero-padding and duplicated token copies. X-MoE removes zero-padding with a sparse token buffer, cuts repeated inter-node traffic by routing only one pilot copy of each token across the network, and shards the sequence inside MoE blocks. On the Frontier supercomputer, the paper reports that X-MoE trains a 545-billion-parameter DeepSeek-style model on 1024 AMD MI250X GPUs, ten times larger than the largest model prior systems could fit under the same hardware budget, at an aggregated 10.44 petaFLOP/s. The implication, if correct, is that the next generation of expert-specialized MoEs can be trained on non-NVIDIA HPC clusters instead of being limited to CUDA-only infrastructure.","feed_headline":"X-MoE trains 545B-parameter MoEs on 1024 AMD GPUs","feed_subtitle":"Padding-free routing and sequence sharding push DeepSeek-style models 10x beyond prior memory limits.","key_machinery":"Three mechanisms carry the argument. The first is a Padding-Free Token buffer (PFT): a sparse layout that stores only the tokens actually routed to each expert, plus small arrays recording token ids, expert ids, per-expert token counts, and combine weights, so no zero-padded capacity slots travel through dispatch, expert MLPs, or combine. The second is redundancy-bypassing dispatch (RBD), a hierarchical two-stage all-to-all that sends one pilot token across a node boundary and reconstructs local replica tokens inside the destination node using fast intra-node bandwidth. The third is sequence-sharded MoE blocks (SSMB): because gating, dispatch, expert FFNs, and combine are token-wise operations, the input sequence is split across expert-parallel ranks inside the MoE block and reassembled with an all-gather before the next tensor-parallel block, cutting the memory of dispatch/combine activations by the tensor-parallel group size. A portable, Triton-based kernel set for gather, scatter, and sequential GEMM lets these sparse stages run on AMD ROCm without CUDA-specific code.","core_discovery":"The paper's central claim is that the two structural changes in emerging MoEs—many more, smaller experts and much larger $k$—move the training bottleneck from parameter memory to activation memory, especially the dispatched-input and combined-output tensors, and that a system built around that shifted bottleneck can make such models trainable where prior systems run out of memory. The evidence is a set of Frontier experiments: X-MoE trains 10.1B, 55.2B, and 201B parameter models on 256 GPUs while the compared baselines run out of memory on all but the smallest, and it reaches 5.15x and 1.42x throughput over the two surviving baselines on the 55.2B model. On 1024 GPUs, X-MoE fits and runs a 545.4B parameter model at 10.44 petaFLOP/s aggregate throughput. The paper also validates correctness by showing that on the 10.1B model X-MoE's loss curve tracks the baseline's over 500 iterations, with a small gap attributed to a more permissive token-dropping rule. The scale claim for the 545B model rests on fitting and throughput rather than a reported loss curve for that run.","pith_inferences":["If the claimed bottleneck shift is general, the pilot-token dispatch trick should also reduce communication in MoE inference on hierarchical networks, since the redundancy it removes is topology-driven rather than training-specific; the paper does not evaluate inference.","A direct test of RBD's mechanism would compare dispatch time on a cluster with balanced high-bandwidth inter-node links: there the redundancy-bypassing scheme should give near-zero benefit, while on an asymmetric topology it should reproduce the reported dispatch speedup.","The absence of a 545B loss curve leaves open whether that run is sustained training or a memory-fit benchmark; reporting the 545B loss trajectory and completed iteration count would settle the distinction.","The paper's placement analysis suggests a testable prediction: on nodes with even higher intra-node bandwidth than Frontier's, a DP-first placement strategy should widen the gap over the EP-first strategy used by existing systems."],"forward_implications":["The trainable boundary for DeepSeek-style MoEs moves to over half a trillion parameters on 1024 GPUs, where prior systems cannot fit even the 55.2B model.","Padding-free routing cuts per-layer activation memory substantially, so sequence length, top-$k$, and expert count can be pushed further before memory limits bind.","On the 55.2B model, X-MoE is faster than the strongest baseline that can fit the model, and its advantage grows with larger top-$k$ values.","Because the kernels use a portable backend rather than CUDA, the same system runs on NVIDIA A100s as well, fitting configurations that CUDA-only baselines OOM on."],"supporting_citations":[{"why":"Introduces the capacity-factor, zero-padded dispatch scheme that PFT replaces and that motivates the padding-free pipeline.","marker":"[24]"},{"why":"DeepSpeed-MoE is the primary baseline and integration base, and its token-dropping logic is compared against in the loss validation.","marker":"[31]"},{"why":"Tutel is the strongest surviving baseline for throughput comparisons and the memory measurements on AMD GPUs.","marker":"[16]"},{"why":"DeepSpeed-TED is the tensor-expert-data parallel baseline whose memory behavior SSMB is compared against analytically and empirically.","marker":"[34]"},{"why":"Triton supplies the portable kernel backend that makes the sparse kernels hardware-agnostic rather than CUDA-specific.","marker":"[36]"},{"why":"Frontier is the evaluated platform, providing the 200 GB/s intra-node versus 25 GB/s inter-node bandwidth asymmetry that motivates RBD and placement choices.","marker":"[5]"},{"why":"The Dragonfly topology model explains the hierarchical inter-node latencies and the cross-rack outlier all-to-alls observed at scale.","marker":"[21]"},{"why":"DeepSeek-MoE defines the fine-grained-expert, large-top-$k$ architecture and supplies the Small model configuration used in experiments.","marker":"[10]"},{"why":"DeepSeek-V3 supplies the 256-expert, top-8 configuration used for the Medium, Large, and Super model evaluations.","marker":"[11]"}],"fun_headline_variants":["X-MoE trains 545B-parameter MoE on 1024 AMD GPUs","X-MoE scales MoE training 10x beyond prior limits","Activations, not weights, now limit MoE training scale","X-MoE: 10x larger MoE models on same GPU budget","MoE training hits 545B params on AMD GPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 545B run is genuine sustained training, not merely a configuration whose parameter buffers fit in memory; the paper validates convergence only at the 10.1B scale and does not show a loss curve or completed iteration count for the 545B run.","fun_headline_variants_meta":{"raw":{"variants":["X-MoE trains 545B-parameter MoE on 1024 AMD GPUs","X-MoE scales MoE training 10x beyond prior limits","Activations, not weights, now limit MoE training scale","X-MoE: 10x larger MoE models on same GPU budget","MoE training hits 545B params on AMD GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3230,"prompt_tokens":1046,"completion_tokens":2184,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":2085}},"tokens_in":662,"tokens_out":2184,"duration_ms":16036,"temperature":1.0,"reasoning_tokens":2085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:12:57.687384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reported 545.4B configuration to a fixed iteration budget with a logged loss curve and compare its slope to the 201B model's; if the run fits in memory but the loss diverges or the run cannot complete iterations, the trainability claim collapses. Counting the number of completed optimizer steps and sustained throughput over those steps would also settle whether the 1024-GPU result is training or a single-step benchmark.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepSpeed-MoE is the primary baseline and integration base, and its token-dropping logic is compared against in the loss validation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Tutel is the strongest surviving baseline for throughput comparisons and the memory measurements on AMD GPUs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepSpeed-TED is the tensor-expert-data parallel baseline whose memory behavior SSMB is compared against analytically and empirically."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Triton supplies the portable kernel backend that makes the sparse kernels hardware-agnostic rather than CUDA-specific."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frontier is the evaluated platform, providing the 200 GB/s intra-node versus 25 GB/s inter-node bandwidth asymmetry that motivates RBD and placement choices."},{"cited_title":"Dally, Steve Scott, and Dennis Abts","cited_arxiv_id":null,"evidence_quote":"The Dragonfly topology model explains the hierarchical inter-node latencies and the cross-rack outlier all-to-alls observed at scale."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepSeek-MoE defines the fine-grained-expert, large-top-$k$ architecture and supplies the Small model configuration used in experiments."}],"review_version":2}