{"id":"c28d0c1e-a8e9-472c-b165-15c4c668ee1b","arxiv_id":"2505.21411","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A group-constrained Mixture-of-Experts routing rule (MoGE) is proposed and demonstrated in Pangu Pro MoE, a 72B/16B-active sparse LLM that reports faster inference on Ascend NPUs.","lead":"This paper introduces Mixture of Grouped Experts (MoGE), a routing rule that forces a token to activate the same number of experts from each predefined group, making device workloads balanced by construction. The authors use it to build Pangu Pro MoE, a 72B-parameter sparse model that reports high inference throughput on Huawei Ascend NPUs while matching or beating some 32B dense models on benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MoGE's hard per-group constraint guarantees IS=0, but the paper never isolates it from confounds: all speedups are versus dense models, and profiling shows expert weight transfer is only ~29% of decode latency, so the throughput claim is not causally established.","rationale":"The reader's conditional verdict is appropriate and my independent read lands on the same weakest assumption. The paper's architectural mechanism is coherent: grouping experts and enforcing a fixed per-group activation count does guarantee an Imbalance Score of zero under the paper's definition in Eq. (3), and this is a genuine structural property rather than a heuristic. However, the transfer from IS=0 to 'balanced computational load' and from there to 'significantly enhancing throughput' is where the argument becomes insecure. There are two related but distinct gaps. First, computational load balance is not the same as wall-clock balance. Equal expert counts per device do not by themselves equalize execution time when per-expert batch compositions differ, grouped GEMM kernels have workload-dependent efficiency, and attention plus communication occupy large fractions of inference time. The paper's own profiling data make this concrete: attention is 30-50% of inference time in Section 4.2.3, GMM is over 50% in high concurrency, and weight transfer is only 29% of decode latency in Section 5.3. Second, even if MoGE improves load balance, no experiment demonstrates that this improvement causes the reported throughput. All throughput tables are MoGE-versus-dense comparisons, and the dense baselines differ in architecture, dimensions, quantization, and kernels. The one claim that directly quantifies MoGE's effect, Figure 4's 'over 50%' reduction in permute/gmm_up execution-time disparity, is not presented with a concrete experiment, table, or error analysis. The reported benchmark quality results are plausible and the model appears competitive, so I would not reject the paper. But the central efficiency claim currently rests on an architectural guarantee plus a confounded system-level comparison, which is exactly the kind of evidence that needs a controlled MoGE-versus-MoE ablation and the release of artifacts. The reader's conditional verdict already captures this, so no verdict change is needed.","tokens_in":31938,"tokens_out":3089,"duration_ms":42546,"concrete_test":"Run the same 72BA16B Pangu Pro MoE configuration (48 layers, hidden 5120, W8A8, same inference kernels, same TP/EP parallelism) with only the routing rule changed: replace per-group K'=1 selection with global Top-8 selection plus the same auxiliary load-balancing loss. Measure decode throughput at batch size 456 on Ascend 800I A2, end-to-end latency, and the execution-time disparity across devices. If throughput drops by less than roughly 10% while IS rises substantially, MoGE's load-balancing constraint is not the driver of the reported 1148 tokens/s and the throughput claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in the paper's central claim is the inference that equal per-device expert counts imply balanced wall-clock load, and that this balance is what produces the reported throughput gains. The first link is asserted rather than demonstrated: Section 2.2 proves IS=0 for MoGE, but IS counts expert activations, not execution time. Devices with the same number of activated experts can still finish at different times because per-expert token counts differ, grouped-GEMM shapes vary, and attention/communication overhead is substantial. The paper's own profiling weakens the causal link: Section 4.2.3 states attention can be 30-50% of total inference time and GMM accounts for over 50% of end-to-end latency in high-concurrency scenarios, while Section 5.3 reports weight transfer is only 29% of decode latency. Thus even a perfect expert-count balance may sit behind a much larger bottleneck. The second link is untested in the experimental design: Table 6 compares the 72BA16B MoGE model against dense 32B/72B models, with different hidden sizes, layer counts, quantization, and custom kernels. No conventional Top-K MoE at matched scale is ever run, so the 97%/18% decode throughput advantages cannot be attributed to MoGE. Figure 4 reports a 'over 50%' reduction in disparity for permute and gmm_up operators, but this is a component-level measurement, not end-to-end, and the baseline details are omitted. The architecture is internally consistent, but the central efficiency claim is not causally supported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mixture of Grouped Experts (MoGE), an MoE variant that partitions experts into groups and forces each token to activate a fixed number of experts from every group, thereby guaranteeing an Imbalance Score (IS) of 0 by construction. The authors build Pangu Pro MoE, a 72B-total/16B-activated MoE trained on 13T tokens and optimized for Ascend NPUs, and report inference throughput of 1148 tokens/s per card on Ascend 800I A2. The paper also describes system-level optimizations (H2P parallelism, MulAttention, SwiftGMM, quantization) and reports benchmark results against dense and MoE baselines. The central claim is that MoGE's group-balanced routing produces better load balance and significantly higher training and inference throughput than conventional MoE.","tokens_in":32327,"tokens_out":4330,"duration_ms":44816,"significance":"If the causal efficiency claim were established, MoGE would be a valuable architectural contribution: it is simple, requires no change to the expert computation itself, and its IS=0 guarantee is mathematically true by construction. The paper also contains substantial engineering content, including a hierarchical hybrid parallel inference scheme, fused kernels, and expert-aware quantization, and it reports a trained 72B model with extensive benchmark numbers. These strengths are real. However, the paper does not yet demonstrate that the MoGE routing constraint is the cause of the reported throughput gains; all efficiency comparisons are against dense models, and the paper's own profiling suggests that expert-count balance is only one component of end-to-end latency. The benchmark claims are also stronger than the data support. The significance is therefore conditional on a controlled ablation and a more careful causal analysis.","major_comments":[{"comment":"The IS=0 guarantee is a statement about expert activation counts per device, not about wall-clock execution time. Section 4.2.3 reports that attention can consume 30%-50% of total inference time and that GMM accounts for over 50% of end-to-end latency in high-concurrency settings, while Section 5.3 reports that weight transfer is only 29% of decode latency. Under these conditions, equal per-device expert counts do not necessarily equalize device finish times, because per-expert token counts, grouped-GEMM shapes, attention, and communication all vary. The paper should provide a measured per-device time breakdown for a MoGE deployment versus a Top-K MoE deployment, or otherwise directly demonstrate that the MoGE constraint is the causal source of the 97%/18% decode throughput advantages reported in Table 6.","section":"Section 2.2, Eq. (3)-(7); Section 5.3; Section 4.2.3"},{"comment":"All inference efficiency comparisons are against 32B and 72B dense models that differ in hidden size, layer count, KV-cache size, communication volume, quantization, and custom kernels. No conventional Top-K MoE with matched total and activated parameter counts is evaluated under the same system optimizations. Without such a baseline, the reported throughput advantages cannot be attributed to MoGE; they could arise from the smaller activated parameter count, the smaller hidden dimension, or the custom Ascend kernels. A same-scale Top-K MoE baseline, ideally with the same H2P parallelism and kernels, is required to support the central efficiency claim.","section":"Section 5.3, Table 6"},{"comment":"The claim that MoGE mitigates computational load imbalance 'by over 50%' is based on reduced maximum disparity in execution time for the permute and gmm_up operators only. The comparison baseline is not described: no configuration, routing scheme, batch size, or measurement procedure is given. This component-level measurement does not establish end-to-end throughput improvement, especially because the same section says attention and GMM dominate different regimes. The paper should either report full end-to-end time distributions for MoGE versus Top-K MoE or temper the claim to the specific operators measured.","section":"Figure 4 caption; Section 4.2.3"},{"comment":"The MoGE argument in Section 2.2 assumes that each expert group is mapped to a distinct device, so equal per-group activation counts imply equal per-device counts. The inference deployment, however, uses TP2+EP4 hybrid parallelism for expert modules, which splits expert matrices across devices and changes the mapping between groups and physical devices. The paper does not explain how the per-group constraint translates into per-device balance under this hybrid parallel scheme. Without this clarification, the theoretical IS=0 guarantee does not directly apply to the reported inference configuration.","section":"Section 4.2.1 (H2P parallelism) versus Section 2.2"},{"comment":"The abstract states that Pangu Pro MoE outperforms Qwen3-32B and GLM-Z1-32B, but Table 4 shows Pangu Pro MoE trailing Qwen3-32B on LiveCodeBench (59.6 vs. 62.6), MBPP+ (80.2 vs. 82.0), AIME2024 (79.2 vs. 80.4), AIME2025 (68.1 vs. 70.9), and MMLU-Redux (81.5 vs. 83.2). The claim of overall outperformance is therefore not supported by the presented numbers. The abstract and conclusion should be revised to describe the results as competitive rather than uniformly superior.","section":"Abstract; Section 5.2.2, Table 4"}],"minor_comments":[{"comment":"The word 'surgence' should be 'surge' or 'resurgence'.","section":"Abstract"},{"comment":"The headings contain a typo: 'Evalutaion' should be 'Evaluation'.","section":"Section 5.1.1, 5.1.2, 5.2.1"},{"comment":"The baseline name appears as 'Llama-4-Scout-baset'; the final 't' appears to be a typo for 'base'.","section":"Table 3"},{"comment":"The symbol α is used for both the auxiliary load-balancing loss weight in Eq. (8) and the quantization migration strength in Eq. (10). These are unrelated hyperparameters and should use different symbols.","section":"Eq. (8) and Eq. (10)"},{"comment":"The training hardware is described as 'Ascend 800T A2', while inference experiments use 'Ascend 800I A2'. The relationship between these platforms should be clarified.","section":"Section 3.1.2"},{"comment":"The caption refers to 'right, blue bars' for the MoE distribution, but the figure shows MoGE as a yellow bar and MoE as blue bars; the color-description mapping should be corrected.","section":"Figure 2(b)"},{"comment":"The MFU improvement is reported only as a relative 35% increase with no absolute MFU values or wall-clock training throughput. Absolute numbers would make the claim more interpretable.","section":"Table 2"},{"comment":"Simulation results are normalized relative to a randomly selected candidate, which makes it difficult to assess absolute throughput or to reproduce the selection of the final configuration. Reporting absolute simulated throughput values would help.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report from a large industrial team, and it combines a genuinely simple architectural idea with a large amount of system engineering. The main gap is experimental: the absence of any same-scale, same-system Top-K MoE ablation means the central efficiency claim is not yet causally supported. The benchmark overclaim in the abstract is also a concern for a public arXiv paper. I believe the manuscript can be revised to address these issues within its scope, hence major_revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Pangu Pro MoE report. The core idea is simple: partition the routed experts into M groups, force each token to pick K' experts from every group, and place one group per device. The imbalance score becomes zero by construction. That part is correct, and as a systems trick it is sensible. What is new is modest — a constrained routing rule plus a full Ascend-oriented system built around it — but the report makes the idea concrete and describes the training pipeline, H2P parallelism, MulAttention, SwiftGMM, and quantization in enough detail that a systems person would learn real things. That is the paper's genuine value.\n\nThe soft spots are the ones the stress-test flags. The throughput claims are not causally attributed to MoGE. Every comparison is against dense 32B/72B models with different hidden sizes, layer counts, quantization, and custom kernels. There is no matched-scale Top-K MoE baseline. The paper's own profiling weakens the link further: attention is 30-50% of end-to-end time, GMM is over 50% at high concurrency, and weight transfer is only 29% of decode latency. Equal expert counts per device might help, but the report never isolates that effect. The 'over 50% disparity reduction' in Section 4.1 is component-level, not end-to-end, and the baseline is not described.\n\nSecond, the abstract overstates the benchmark story. Table 4 shows a competitive model, not a clear winner: Pangu Pro MoE beats Qwen3-32B on MMLU-Pro, C-Eval, CMMLU, GPQA-Diamond, SuperGPQA, and MATH-500, but loses on LiveCodeBench, MBPP+, AIME2024, CNMO2024, MMLU-Redux, and Arena-Hard. Several ablations are internal and undisclosed. No error bars and no artifacts are provided.\n\nThird, the IS=0 guarantee is an identity, not a finding. It shows equal activation counts per group, which is true by definition. It does not show equal wall-clock time; that link is asserted, not measured.\n\nThe architecture is internally consistent and the paper is coherent on its own terms. It is a systems technical report, not a hypothesis-testing paper. The audience is MoE systems researchers, especially those working with expert parallelism or Ascend hardware. A survey of load-balanced routing should cite it. It deserves a serious referee, but that referee should require a matched MoE baseline and an end-to-end breakdown before the throughput claims are accepted.\n\nRecommendation: engage with it, but treat the headline numbers as vendor-reported until the ablation exists.","headline":"MoGE is a simple, correct routing constraint, but the throughput claims are vendor-reported until a matched MoE ablation isolates the effect.","tokens_in":32889,"tokens_out":3339,"would_cite":true,"duration_ms":41037,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pangu Pro MoE claims that routing each token to an equal number of experts per predefined group makes device-level load imbalance in MoE vanish by construction, and that the resulting 72B/16B model reaches 1148 tokens/s per card on Ascend…","keywords":["Mixture of Grouped Experts","Mixture of Experts","load balancing","expert parallelism","sparse activation","inference throughput","Ascend NPU","speculative decoding"],"falsifier":"Deploy Pangu Pro MoE's exact 64-expert, 8-group configuration and a standard global Top-K MoE with identical total parameters, active-expert count, width, depth, data, and hardware, then measure per-card decode throughput and the wall-clock spread across the four accelerators for one decoding step. MoGE predicts near-zero inter-device spread on the expert operators for every batch, while the Top-K baseline predicts a spread that grows as batch size shrinks; if the standard MoE matches the throughput under the same 100 ms budget, or the measured MoGE spread is not dramatically smaller, the IS=0 guarantee is not what drives the reported speedups. A cheaper check already sits in the paper's data: the reported >50% reduction in maximum execution-time disparity for the permute and gmm_up operators is an operator-level, not end-to-end, demonstration.","tokens_in":31787,"feed_emoji":"⚖️","tokens_out":16144,"duration_ms":138097,"temperature":0.7,"pith_summary":"Mixture-of-experts models activate only a fraction of their parameters per token, but the fraction varies wildly: popular experts get overloaded and their devices stall the whole pipeline. This paper claims the imbalance can be removed at the architectural level rather than patched with heuristics. Its Mixture of Grouped Experts (MoGE) partitions the expert pool into groups, assigns each group to a device, and forces every token to activate the same number of experts from each group, making the load-imbalance score exactly zero for every token and every batch. The authors build Pangu Pro MoE on this design — 72B total parameters, 16.5B activated per token, trained on 13 trillion tokens — and report 1148 tokens/s per card within a 100 ms latency budget on Ascend 800I A2 hardware, with benchmark scores that match or beat 32B-scale open models while activating roughly half their parameters. A sympathetic reader would care because the paper makes a structural promise: balanced execution comes from the routing rule itself, not from hoping the router learns to behave, which would let MoE deployments skip load-balancing heuristics and over-provisioning.","feed_headline":"Grouped experts zero out device imbalance in MoE","feed_subtitle":"Pangu Pro MoE, 72B total with 16B active, hits 1148 tokens/s per card on Ascend NPUs.","key_machinery":"The load-bearing object is the MoGE routing rule: a global softmax over all $N$ experts followed by an independent Top-$K'$ selection inside each of $M$ predefined expert groups, with the per-group selections concatenated into the final gating vector (Eq. 6). Because each group is mapped to a distinct device, this rule converts load balance from an emergent property into a deterministic constraint — every device handles exactly $K'$ expert computations per token, so the imbalance score is identically zero. Pangu Pro MoE runs the sharpest version of the rule with $K' = 1$: 64 routed experts arranged in 8 groups of 8, one expert drawn from each group, plus 4 shared experts, for 16.5B activated parameters. An auxiliary load-balancing loss is retained, but it is computed from the global softmax scores rather than per-group, so it only shapes usage within groups; inter-group balance no longer depends on the loss at all.","core_discovery":"The paper's central claim is that group-balanced routing suffices to eliminate device-level load imbalance in distributed MoE execution. Formally, with $N$ experts partitioned into $M$ groups and $K' = K/M$ experts activated per group per token, the Imbalance Score $IS(X) = (\\max_i T_i(X) - \\min_i T_i(X))/|X|$ is zero for every batch, whereas Monte Carlo simulation of standard global Top-K routing shows nearly a probability-one chance of nonzero imbalance at small batch sizes. The authors establish the point by construction and then demonstrate it empirically: expert activation in the first MoE layer of Pangu Pro MoE sits near the theoretical 12.5% per expert, against up to 30% for the busiest DeepSeek-V2 expert, and the maximum execution-time disparity of the permute and gmm_up operators drops by over 50%. On the strength of this balance they report that Pangu Pro MoE, a 72B-parameter model activating 16.5B per token, reaches 1148 tokens/s per card in high-concurrency decode on Ascend 800I A2 (1528 tokens/s with multi-token-prediction decoding) and outperforms dense 32B and 72B models under the same latency constraints, while matching or beating open 32B-scale models on many benchmarks.","pith_inferences":["Editorial inference: the IS=0 guarantee matters most exactly where Top-K imbalance is worst — small batches and skewed routing distributions — which is also the regime where the paper's own profiling shows attention and weight transfer dominating wall-clock time, so the routing rule is a necessary but not sufficient condition for the reported speedups.","Editorial inference: because the paper never ablates MoGE against a standard Top-K MoE at matched scale, the causal link from the group constraint to the measured throughput is not isolated; a fair test would hold total parameters, active parameters, data, and hardware fixed and vary only the routing rule.","Editorial inference: MoGE trades routing freedom for a hard balance guarantee — when a token's best experts concentrate in one group, it is forced to take weaker experts elsewhere — so the design implies a quality-versus-balance trade-off curve that the paper does not map, especially at larger $K'$."],"forward_implications":["In any deployment that pins expert groups to devices, every device receives exactly the same number of expert computations per token, so the straggler bottleneck disappears at every batch size, including single-token batches.","Pangu Pro MoE reaches 1148 tokens/s per card (1528 tokens/s with multi-token-prediction decoding) on Ascend 800I A2 within a 100 ms latency budget, exceeding 32B and 72B dense models by 18% and 97% respectively on the same hardware.","On the training side, combining MoGE with hierarchical expert-parallel communication and pipeline overlap raises Model FLOPs Utilization by 35% over the baseline configuration.","Expert utilization becomes nearly uniform — around 12.5% of tokens per expert in the first layer versus up to 30% for the busiest DeepSeek-V2 expert — so the model makes fuller use of its expert capacity.","On the cheaper Ascend 300I Duo, the same model sustains 201 tokens/s per card (321 tokens/s with MTP) within 100 ms latency, which the authors point to as a favorable cost-to-performance ratio for sub-100B inference."],"supporting_citations":[{"why":"supplies the Top-K sparsely-gated routing formalism (Eq. 2) that MoGE replaces with group-balanced selection.","marker":"[30]"},{"why":"documents the expert load-imbalance and automatic-sharding problem in distributed giant models that MoGE is designed to eliminate.","marker":"[18]"},{"why":"serves as the comparison MoE whose global expert distribution (up to 30% of tokens on one expert) is the imbalance baseline in Figure 10.","marker":"[24]"},{"why":"provides the hierarchical expert-parallel communication and pipeline-overlap system techniques that Pangu Pro MoE refines for training efficiency.","marker":"[34]"},{"why":"one of the open 32B-scale baselines that Pangu Pro MoE must outperform to support its leading-sub-100B claim.","marker":"[11]"},{"why":"the other principal 32B dense baseline for both the benchmark comparison (Table 4) and the inference throughput comparison (Tables 5-6).","marker":"[43]"}],"fun_headline_variants":["Grouped routing eliminates MoE device imbalance","MoGE: equal experts per group, zero load imbalance","Pangu Pro MoE: 72B with balanced routing hits 1148 tok/s","Balanced expert selection ends device-level bottleneck","Group-balanced experts improve MoE inference throughput"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported speedups assume expert computation is the dominant and roughly uniform cost across the devices that hold the experts, so that forcing equal per-device expert counts forces equal wall-clock time; the paper never compares MoGE against a standard MoE at the same scale to verify that the routing rule itself causes the gains, and its own profiling puts attention, KV-cache traffic, and communication at large shares of inference time (Sections 4.2.3–4.2.4).","fun_headline_variants_meta":{"raw":{"variants":["Grouped routing eliminates MoE device imbalance","MoGE: equal experts per group, zero load imbalance","Pangu Pro MoE: 72B with balanced routing hits 1148 tok/s","Balanced expert selection ends device-level bottleneck","Group-balanced experts improve MoE inference throughput"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1831,"prompt_tokens":1195,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":811,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":811,"tokens_out":636,"duration_ms":6394,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:28:17.764989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy Pangu Pro MoE's exact 64-expert, 8-group configuration and a standard global Top-K MoE with identical total parameters, active-expert count, width, depth, data, and hardware, then measure per-card decode throughput and the wall-clock spread across the four accelerators for one decoding step. MoGE predicts near-zero inter-device spread on the expert operators for every batch, while the Top-K baseline predicts a spread that grows as batch size shrinks; if the standard MoE matches the throughput under the same 100 ms budget, or the measured MoGE spread is not dramatically smaller, the IS=0 guarantee is not what drives the reported speedups. A cheaper check already sits in the paper's data: the reported >50% reduction in maximum execution-time disparity for the permute and gmm_up operators is an operator-level, not end-to-end, demonstration.","supporting_citations":[{"cited_title":"Pangu ultra moe: How to train your big moe on ascend npus, 2025","cited_arxiv_id":null,"evidence_quote":"provides the hierarchical expert-parallel communication and pipeline-overlap system techniques that Pangu Pro MoE refines for training efficiency."}],"review_version":1}