REVIEW 3 major objections 5 minor 38 references
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper argues that Mixture-of-Experts language-model evolution is best read as a dependency graph of eight milestones and a closed loop of four control planes, whose main trend is decoupling semantic routing, computational budgets, and…
desk verdict Useful MoE organizing framework, but the headline decoupling trend leans on unverifiable frontier citations and an abstract that promises experiments it doesn't run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the four-control-plane closed loop: Topology defines the expert set, Routing selects an expert subset per token, Balance aggregates load statistics and feeds back loss, bias, capacity, placement, or replication, and Expert Parallelism maps the selection to dispatch, local GEMM, and combine on physical devices. The paper formalizes the loop as $\mathcal{E}=T(\theta_{\mathrm{topo}})$, $\mathcal{K}_t=R(x_t,\mathcal{E};\theta_{\mathrm{route}},b)$, $(b,\alpha,c)\leftarrow C(\{n_i\},\pi)$, and $y_t=\mathrm{EP}(x_t,\mathcal{K}_t,\pi,\sigma)$, showing that the four layers are not independent modules but a closed system. The companion object is the eight-milestone dependency graph, which records which bottleneck each historical change removed and what bottleneck it exposed, so the control planes and the milestone graph serve as complementary temporal and structural views.
What would settle it
Run a controlled equal-budget pretraining sweep, same data, token budget, and active parameters, varying expert granularity, shared path, balance scope, and execution topology independently, and measure validation loss, specialization, exposed communication, and tail latency; if no control-plane knob changes the outcomes, the bottleneck-migration story would be refuted.
Extended reading notes
Core claim
The paper's central claim is a re-description of MoE history and structure. Chronologically, it proposes eight architectural milestones tied to bottleneck migration: statistical division of labor, sparse Top-k conditional computation, Transformer-scale expert parallelism, open decoder-only MoE, fine-grained and shared experts, ultra-sparse scaling, token-dependent compute budgets, and semantic routing decoupled from physical execution. These are presented as a dependency graph, with nodes 1 through 6 forming a mainline and nodes 7 and 8 as stackable orthogonal branches, not as eight successive generations. Structurally, the paper claims that every MoE system is a closed loop of four control planes, Topology, Routing, Balance, and Expert Parallelism, with forward execution from topology to routing to execution and feedback from load statistics and system costs back to routing and placement. The paper's principal conclusion is that the main trend across this history is a shift from activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.
Load-bearing premise
The framework stands on the accuracy of third-party technical reports the author did not independently verify, so a misdescribed architecture or reported scaling number would loosen the empirical support for the milestone pattern and the claimed macro-trend.
Editorial extensions
If this is right
- Comparing MoE systems by release date or benchmark ranking becomes less informative than locating them on the four control planes and tracing their bottleneck migration.
- The modern mainline, token-choice Top-k routing with fine-grained experts, optional shared path, global or ALF balance, topology-limited routing, and runtime expert placement, is predicted to persist while frontier branches evolve as combinations rather than replacements.
- Dynamic compute and semantic-physical decoupling are stackable with fine-grained and ultra-sparse structures, so there is no single next-generation architecture after ScMoE.
- MoE evaluation should report active budget, exposed communication, specialization metrics, and tail latency, not just validation loss, because average training FLOPs do not guarantee systems efficiency.
- Load balancing should be decomposed into expert, device, node, and communication levels, with runtime placement and replication treated as part of the balance plane rather than as post-hoc engineering.
Reading between the lines
- If the framework holds, the same four-plane decomposition could be used to compare MoE designs outside decoder-only pretraining, such as code models or multimodal systems, and to identify which control-plane knob produces a reported gain.
- A testable extension is to adopt the paper's proposed equal-budget ablation matrix, fixed data, tokens, and active FLOPs with granularity, shared path, balance scope, and execution topology varied independently, as a standard protocol for architecture papers.
- The claimed decoupling trend suggests that load balancing will migrate further from training-time losses toward runtime expert placement and replication, making routing stability under domain shift a more central research problem than new router families.
- If the bottleneck-migration story is correct, future innovations should be sought where current bottlenecks concentrate: exposed communication, small-GEMM fragmentation, expert-weight I/O, and tail-latency variance under dynamic compute.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This technical survey proposes a framework for organizing the evolution of Mixture-of-Experts (MoE) architectures in large language models. It introduces eight architectural milestones arranged as a dependency graph (six mainline developments, two orthogonal branches) and four control planes (Expert Topology, Routing, Balance, Expert Parallelism). The paper formalizes token-choice MoE, the capacity/quality/system-efficiency trade-off, and the closed-loop relations among the control planes in Equations (1) through (7). It applies this framework to representative models and concludes that the main trend is a shift from activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.
Significance. If the framework is accepted, it offers a useful organizing schema for comparing MoE designs and tracing bottleneck migration. The mathematical formalizations are standard and correct, and the paper credibly separates algorithmic issues (routing, granularity, sharing) from systems issues (all-to-all communication, expert placement, overlap). Concrete strengths include a comprehensive architectural comparison table (Table 6), a practical equal-budget ablation design (Table 7), and a clear articulation of open research questions. However, the manuscript's empirical support is limited: Section 9 contains only a proposed experimental matrix with no results, and the macro-trend claim in Section 11 rests substantially on sources that are either unverifiable or explicitly acknowledged as lacking public validation. These issues affect the paper's central claim rather than peripheral presentation.
major comments (3)
- [Abstract and Section 9] The abstract states 'We conclude with equal-budget pretraining experiments,' but Section 9 presents only a proposed experimental design (Table 7) and reports no experimental outcomes. The phrasing in the abstract implies that actual equal-budget experiments were run. Either conduct and report the experiments, or revise the abstract and Section 11 to state that the paper provides an experimental design for future work. This is load-bearing because the macro-trend conclusion is framed as an observed evolution rather than a hypothesis.
- [Sections 3.7, 3.8, 8, 11 and Table 6] The central claim that the main trend is decoupling semantic routing, computational budgets, and physical execution rests on nodes 7 and 8. These nodes are supported by references [19], [21], [22], [23], [24], and [26]. Several of these citations are unverifiable as given: [22] lists 'MoHGE Authors' as author, [23] lists 'GMoE Authors', [26] lists 'Expert Upcycling Authors', and [21] is a single blog URL with no archival record. The paper itself concedes in Section 3.8 and Section 8 that these branches lack public validation at trillion-parameter scale and should be treated as a research frontier. Given this concession, the strong conclusion in Section 11 ('the main trend is a shift...') is not supported by the evidence presented. Please either verify and properly archive these sources, or reframe the claim as a projection based on frontier research directions.
- [Section 3, milestone criteria] The paper defines a milestone as a change that satisfies three criteria, the third being that 'later architectures inherit the change.' However, nodes 7 and 8 are explicitly described as orthogonal branches with no clear evidence of inheritance, and the paper itself says they are not 'generations' following node 6. This creates an internal inconsistency: nodes 7 and 8 are called milestones while failing the paper's own inheritance criterion. Please adjust the criteria to accommodate branches (e.g., 'influence future design' rather than 'inheritance'), or rename the components to distinguish established milestones from frontier branches.
minor comments (5)
- [Throughout] The manuscript contains numerous typographical errors and inconsistent spelling, e.g., 'official' in the Abstract, 'efficient' and 'sufficient' in multiple sections, and inconsistent use of 'Chapter 3' versus 'Section 3'.
- [Section 9] The opening sentence of Section 9 is an incomplete fragment: 'cross-model benchmark cannot identify architectural contributions.' Please provide a subject and connect it to the following sentence.
- [References] Several references are non-archival or use placeholder-like author names. For example, [14] and [21] are cited as 'official technical blog' and 'Technical Blog' with only a URL; [22], [23], and [26] use 'Authors' as the author field. Please add access dates, archive links (e.g., DOI or persistent repository), and full author lists where available.
- [Equation (7), Section 6.3] The bias update rule b_i ← b_i + η sign(n̄ − n_i) is described as non-gradient, but it is not clear how η is selected or whether this update interacts with the routing gradient through the Top-k selection. A one-sentence clarification of the design rationale and stability considerations would be helpful.
- [Appendix] The manuscript includes a full Chinese translation of the paper after the reference list. This duplication is unusual for a journal submission and should be moved to supplementary material or removed, as it distracts from the main text.
Circularity Check
No circular derivation: the paper is a survey that classifies existing MoE systems, and its trend claim is an interpretive synthesis, not an equation-level fit or self-citation reduction.
full rationale
This is a survey and synthesis paper rather than a derivation, so the circularity checks for fitted parameters, predicted quantities, uniqueness theorems, or equation-level equivalences do not apply. Its chain is: define five coupled dimensions and four control planes (Section 2), classify existing systems, identify eight milestones using three explicit criteria (new adjustable design variable, bottleneck migration, and inheritance by later architectures; Section 3), and then state a macro-trend in Section 11. None of these steps fits a parameter or predicts a numerical result from a fit. Equations (1)-(7) are standard formalizations of token-choice MoE, capacity approximations, load variation, balance losses, and ALF bias; they are not used to derive the milestone taxonomy or the final trend. The milestone criteria are stated before the classification and do not mention the conclusion, so the taxonomy is not definitionally forced to produce the claimed trend; it is a classification judgment about existing systems. The paper also explicitly flags the uncertain branches: Section 3.8 says that in the absence of public validation at trillion-parameter and ten-thousand-accelerator scale, those branches should be treated as a research frontier, not as a mainstream replacement, and Section 8 lists several branches as not yet having reached a unified conclusion. The main weakness identified by the skeptic is that nodes 7-8 rest on third-party reports with placeholder author lists ([22], [23], [26], etc.) and unverifiable arXiv IDs; that is an external-validation or sourcing concern, not a circularity concern, because the cited sources are not the present author's own fitted inputs and the paper does not assert their claims as derived results. No load-bearing self-citation chain is identifiable from the text: ScMoE is cited as prior work [20], the paper explicitly notes that ScMoE is not a new routing algorithm, and the text does not establish that the present author is an author of [20]. Therefore there is no significant circularity.
Assumptions & free parameters
assumptions (3)
- standard math Standard token-choice MoE equations (Equations 1 and 2) define the computational model.
- ad hoc to paper The four control planes (Topology, Routing, Balance, Expert Parallel) and their closed-loop relation (Equation 5) are a valid abstraction of MoE systems.
- domain assumption The cited technical reports and papers accurately describe the systems' architectures and numbers.
Cite this review
Pith. "Pith review of The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism." pith.science (2026). https://pith.science/paper/QXPUTT2Z
@misc{pith2026260808650,
author = {Pith},
title = {Pith review of: The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism},
year = {2026},
howpublished = {\url{https://pith.science/paper/QXPUTT2Z}},
note = {Machine review of arXiv:2608.08650}
}
read the original abstract
Mixture-of-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone. This technical survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure. We describe eight architectural milestones as a dependency graph with six mainline developments and two orthogonal branches, rather than as eight successive generations. We then analyze individual systems through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism. These planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation is mapped onto physical devices. The framework connects algorithmic choices such as Top-k routing, shared experts, fine-grained experts, and dynamic expert composition with systems concerns including token dispatch, device placement, all-to-all communication, and communication-computation overlap. We conclude with equal-budget pretraining experiments, quality and systems metrics, and open research questions. The main trend is a shift from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.
Reference graph
Works this paper leans on
-
[37]
限制通信拓扑：device/node-limited routing 降低 fan-out；
-
[38]
隐藏通信：shared computation overlap、DualPipe、ScMoE 和 token chunking 扩大 overlap window；
-
[39]
重构通信：Head Parallel 等方案把动态流量变为更确定的通信模式。 这里必须区分“总 All-to-All 时间”和“exposed communication time ”。若通信完全被 dense path 覆盖，继 续优化链路带宽对端到端 step time 的边际收益会下降；反之，即使总通信量不变，只要关键路径缩短，吞吐也可 25 LLM MoE 架构演进 J 开放问题 能显著改善。因此训练报告至少应同时给出通信量、overlap 比例、exposed dispatch/combine、MFU 和 step-time P99。 H 现代主干与前沿分支 截至 2026 年，大规模 decoder MoE 最常见的组合可概括为： Token-choice Top-𝑘 + fine-grain...
work page 2026
-
[40]
A Survey on Mixture of Experts in Large Language Models
Cai W, et al. A Survey on Mixture of Experts in Large Language Models. IEEE Transactions on Knowledge and Data Engineering,
-
[41]
https://arxiv.org/abs/2407.06204
-
[42]
A Survey on Inference Optimization Techniques for Mixture of Experts Models
Liu J, et al. A Survey on Inference Optimization Techniques for Mixture of Experts Models. arXiv:2412.14219, 2024. https://arxiv.org/abs/2412.14219
arXiv 2024
-
[43]
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
Zhu X, et al. Speed Always Wins: A Survey on Efficient Architectures for Large Language Models. arXiv:2508.09834, 2025. https://arxiv.org/abs/2508.09834
arXiv 2025
-
[44]
Adaptive Mixtures of Local Experts
Jacobs R A, Jordan M I, Nowlan S J, Hinton G E. Adaptive Mixtures of Local Experts. Neural Computation, 3(1):79–87, 1991. doi:10.1162/neco.1991.3.1.79
Show all 38 references
-
[45]
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer N, et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR, 2017. https://arxiv.org/abs/1701.06538
2017 arXiv
-
[46]
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin D, et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ICLR, 2021. https://arxiv.org/abs/2006.16668
2021 arXiv
-
[47]
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus W, Zoph B, Shazeer N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. https://arxiv.org/abs/2101.03961
2022 arXiv
-
[48]
ST-MoE: Designing Stable and Transferable Sparse Expert Models
Zoph B, et al. ST-MoE: Designing Stable and Transferable Sparse Expert Models. arXiv:2202.08906, 2022. https://arxiv.org/abs/2202.08906
2022 arXiv
-
[49]
BASE Layers: Simplifying Training of Large, Sparse Models
Lewis M, et al. BASE Layers: Simplifying Training of Large, Sparse Models. ICML, 2021. https://arxiv.org/abs/2103.16716
2021 arXiv
-
[50]
Mixture-of-Experts with Expert Choice Routing
Zhou Y, et al. Mixture-of-Experts with Expert Choice Routing. NeurIPS, 2022. https://arxiv.org/abs/2202.09368
2022 arXiv
-
[51]
Mixtral of Experts
Jiang A Q, et al. Mixtral of Experts. arXiv:2401.04088, 2024. https://arxiv.org/abs/2401.04088
2024 arXiv
-
[52]
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Dai D, et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. ACL, 2024. https://arxiv.org/abs/2401.06066
2024 arXiv
-
[53]
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024. https://arxiv.org/abs/2405.04434
2024 arXiv
-
[54]
Arctic: Snowflake’s Open-Source LLM
Snowflake. Arctic: Snowflake’s Open-Source LLM. Official Technical Blog, 2024. Accessed 2026-08-07. https://www.snowflake.com/en/blog/arctic-open-efficient-foundation-language-models-snowflake/
2024
-
[55]
Qwen3 Technical Report
Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025. https://arxiv.org/abs/2505.09388
2025 arXiv
-
[56]
Mixture of A Million Experts
He X. Mixture of A Million Experts. arXiv:2407.04153, 2024. https://arxiv.org/abs/2407.04153
2024 arXiv
-
[57]
Kimi K2: Open Agentic Intelligence
Moonshot AI. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025. https://arxiv.org/abs/2507.20534
2025 arXiv
-
[58]
gpt-oss Model Card: Architecture
OpenAI. gpt-oss Model Card: Architecture. 2025. Accessed 2026-08-07. https://deploymentsafety.openai.com/gpt-oss/architecture
2025
-
[59]
LongCat-Flash Technical Report
LongCat Team. LongCat-Flash Technical Report. arXiv:2509.01322, 2025. https://arxiv.org/abs/2509.01322
2025
-
[60]
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
Cai W, et al. Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts. arXiv:2404.05019, 2024. https://arxiv.org/abs/2404.05019
2024 arXiv
-
[61]
LongCat 2.0 Technical Blog
LongCat Team. LongCat 2.0 Technical Blog. 2026. Accessed 2026-08-07. https://longcat.ai/blog/longcat-2.0/
2026
-
[62]
MoHGE: Mixture of Heterogeneous Grouped Experts
MoHGE Authors. MoHGE: Mixture of Heterogeneous Grouped Experts. ACL Industry Track, 2026. https://aclanthology.org/2026.acl-industry.20/
2026
-
[63]
GMoE: Global Mixture-of-Experts
GMoE Authors. GMoE: Global Mixture-of-Experts. ACL, 2026. https://aclanthology.org/2026.acl-long.2065/
2026
-
[64]
Multi-Head LatentMoE and Head Parallel
Cui Y, et al. Multi-Head LatentMoE and Head Parallel. arXiv:2602.04870, 2026. https://arxiv.org/abs/2602.04870
2026
-
[65]
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Komatsuzaki A, et al. Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. arXiv:2212.05055, 2022. https://arxiv.org/abs/2212.05055
2022 arXiv
-
[67]
https://arxiv.org/abs/2604.19835
-
[68]
From Sparse to Soft Mixtures of Experts
Puigcerver J, et al. From Sparse to Soft Mixtures of Experts. ICLR, 2024. https://arxiv.org/abs/2308.00951
2024 arXiv
-
[69]
Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
Zhong Z, et al. Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training. COLM, 2024. https://arxiv.org/abs/2405.03133. 29 LLM MoE 架构演进 参考文献
2024 arXiv
-
[70]
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
Wang Q, et al. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. arXiv:2408.15664, 2024. https://arxiv.org/abs/2408.15664
2024 arXiv
-
[71]
DeepSeek-V3 Technical Report
DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024. https://arxiv.org/abs/2412.19437
2024 arXiv
-
[72]
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
Gale T, et al. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. MLSys, 2023. https://arxiv.org/abs/2211.15841
2023 arXiv
-
[73]
Tutel: Adaptive Mixture-of-Experts at Scale
Hwang C, et al. Tutel: Adaptive Mixture-of-Experts at Scale. MLSys, 2023. https://arxiv.org/abs/2206.03382
2023 arXiv
-
[74]
JetMoE: Reaching Llama2 Performance with 0.1M Dollars
Shen Y, et al. JetMoE: Reaching Llama2 Performance with 0.1M Dollars. arXiv:2404.07413, 2024. https://arxiv.org/abs/2404.07413
2024 arXiv
-
[75]
Jamba: A Hybrid Transformer–Mamba Language Model
Lieber O, et al. Jamba: A Hybrid Transformer–Mamba Language Model. arXiv:2403.19887, 2024. https://arxiv.org/abs/2403.19887. 30
2024 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.