MH-MoE: Multi-Head Mixture-of-Experts

Furu Wei; Shaohan Huang; Shuming Ma; Xun Wu

REVIEW 4 major objections 4 minor 17 references

MH-MoE: Multi-Head Mixture-of-Experts

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-head mixture-of-experts layer can be configured to match sparse MoE in FLOPs and parameters while improving language-model perplexity.

desk verdict The practical recipe is appealing, but the FLOPs-parity derivation has an internal factor-of-h error in the activated-expert cost, so the central claim is unsupported as written. read the letter →

arxiv 2411.16205 v3 pith:GD7R4NKT submitted 2024-11-25 cs.CL

Shaohan Huang , Xun Wu , Shuming Ma , Furu Wei This is my paper · ORCID

classification cs.CL

keywords mixtureofexpertsmulti-headMoEFLOPsparitysparselanguagemodelingBitNetTransformerperplexity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-Head Mixture-of-Experts has previously shown quality gains, but its original parameter choices cost more FLOPs than a vanilla sparse MoE layer. This paper tries to establish that the gains can be kept without paying extra compute by shrinking each expert's intermediate dimension by $d/k$ and adding experts to restore the parameter count. In decoder-only Transformer pretraining, the resulting MH-MoE variants consistently obtain lower validation perplexity than both vanilla and fine-grained sparse MoE on the RedPajama, Wiki, and C4 validation sets at matched parameters and computation. The same ordering holds when all models are trained with 1-bit BitNet quantization. If the paper is right, multi-head routing is a drop-in, compute-neutral replacement for standard sparse MoE layers in language models.

What carries the argument

The load-bearing object is the head/merge projection pair around a shrunken MoE layer. The input is projected by $W_{\mathrm{head}}$, split into $h$ sub-token views of dimension $d/h$, passed through top-$k$ experts of intermediate dimension $d_{\mathrm{mhmoe}}$, concatenated, and projected by $W_{\mathrm{merge}}$. Equation 8, $d_{\mathrm{mhmoe}} = d_{\mathrm{moe}} - d/k$, sets the intermediate dimension so the first-order FLOP count of the whole layer equals the standard SMoE count $4B d d_{\mathrm{moe}} k$, and Equation 9 derives the required expert count by matching $2 d d_{\mathrm{moe}} E_{\mathrm{moe}}$ to $2d^2 + 2 (d/h) d_{\mathrm{mhmoe}} E_{\mathrm{mhmoe}}$. The argument works because the two extra projections cost $4Bd^2$ in the leading term, exactly the amount the expert shrink removes.

What would settle it

Recompute the exact multiply counts for the reported configurations (e.g., head=3, $d=768$, $d_{\mathrm{mhmoe}}=512$, top-3) including the head and merge projections; if the leading term is not the same as the baseline SMoE, the parity claim fails. A simpler empirical check is to time a training step of the MH-MoE model against the vanilla SMoE baseline at equal batch size; similar wall-clock time supports the claim, while a large gap would contradict it.

Watch

Extended reading notes

Core claim

The paper claims that the correct sizing of a multi-head MoE layer makes its compute exactly equal to a standard sparse MoE layer. Given a baseline with intermediate dimension $d_{\mathrm{moe}}$ and $k$ activated experts, setting $d_{\mathrm{mhmoe}} = d_{\mathrm{moe}} - d/k$ and increasing the expert count through Equation 9 keeps the leading FLOP term and the parameter count fixed. With this recipe, two configurations, head=2 with top-2 gating and 40 experts and head=3 with top-3 gating and 96 experts, beat the dense baseline, a vanilla 8-expert SMoE, and a fine-grained 16-expert SMoE at every reported checkpoint and validation dataset. The head-3 configuration is the strongest throughout, and the improvement persists in the 1-bit BitNet setting.

Load-bearing premise

The paper's equal-compute claim rests on its FLOPs count for the MH-MoE layer: the activated-expert term is written as $4Bd d_{\mathrm{mhmoe}} k$ with no division by the head count $h$, even though each expert sees a $d/h$-dimensional input; if that count is wrong for the SwiGLU experts used in the experiments, the configurations are not actually FLOPs-matched.

Editorial extensions

If this is right

An existing sparse MoE model can be converted to MH-MoE without changing parameter count by applying Equation 8 and Equation 9, making the perplexity comparisons structurally fair.
With compute held fixed, more heads and more activated experts improve quality: the three-head, top-3 variant beats the two-head, top-2 variant on every reported dataset.
The recipe composes with shared-expert settings such as DeepSeek-style residual experts, where the MH-MoE advantage over fine-grained SMoE remains.
Under BitNet 1-bit training, MH-MoE still outperforms SMoE, indicating multi-head routing helps even when weights are aggressively quantized.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims, the identity $d_{\mathrm{mhmoe}} = d_{\mathrm{moe}} - d/k$ suggests a family of FLOPs-neutral configurations beyond the two tested ones; any head count $h$ can be paired with $k$ activated experts as long as the formula is respected, which is a direct test of whether the quality gain scales with $h$.
Beyond the paper's claims, because the FLOPs argument counts multiply-accumulate operations, actual on-chip parity also depends on memory traffic and kernel efficiency of the two projections and the wider expert set; measuring wall-clock step time is a corollary the paper leaves implicit.
Beyond the paper's claims, the head layer's larger ablation gain suggests the benefit may come from creating multiple routed views before gating rather than from merging after experts; a dedicated study varying only the head projection would clarify the mechanism beyond the paper's reported ablations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one if you want a clean example of why FLOPs accounting matters. The new thing here is a recipe—Eq. 8 and 9—for setting MH-MoE's expert width, expert count, and top-k so that the model matches a standard SMoE in FLOPs and parameters, plus a BitNet compatibility check. If that recipe worked, it would be a modest, useful result for MoE practitioners. It probably doesn't, as written.

The paper does some things well. The ablations are clean: adding head and merge layers to a vanilla SMoE gives almost nothing, while the same layers in MH-MoE give a real drop in perplexity, and the head layer does most of the work. The empirical comparisons cover three datasets, two training horizons, and a shared-expert setting, and the MH-MoE numbers are consistently lower than both baselines. That consistency is worth something even at 768 hidden and 100k steps. The BitNet section is a bonus and the result there is also consistent.

The soft spot is the complexity analysis, and it's load-bearing. Eq. 5 defines each expert as a two-matrix FFN with input dimension d/h, so one expert costs 2B d dmoe scalar multiplications per token, and with k experts the leading term should carry a factor 1/h. Eq. 6 writes the activated-expert term as (4B d dmoe - B d - B dmoe h)k with no 1/h and a factor of 2 that is never justified. Eq. 8 is algebraically fine given Eq. 6's leading terms—d_mhmoe = d_moe - d/k does follow—but since Eq. 6 conflicts with the paper's own expert definition, the parity recipe is unsupported. Plugging the Table 1 configurations into the paper's own Eq. 6 gives MH-MoE about 7.08M B against SMoE's 6.29M B, a 12.5% gap, for both h=2 and h=3; parameter parity is off by 1.6% and 4.7%. And all experiments use SwiGLU FFNs, while the derivation assumes ReLU, so the count doesn't transfer directly either.

A few smaller things: the gains are small and there are no error bars or seeds; the baselines' routing details are underspecified; and the paper leans heavily on the authors' own MH-MoE and BitNet papers. None of those is disqualifying by itself. The derivation is.

My take: the empirical pattern might survive a corrected accounting, but the advertised mechanism is not established. I'd send it to a serious referee because the problem is checkable and the experiments are cheap to reproduce, but my expectation would be rejection unless the authors rework the complexity model and re-check parity. I wouldn't cite it in its current form, though I might bring it to a reading group as a cautionary example of why cost models have to match the architecture.

Referee Report

4 major / 4 minor

Summary. The paper presents a new implementation of Multi-Head Mixture-of-Experts (MH-MoE) that is claimed to keep both FLOPs and parameter parity with standard sparse Mixture-of-Experts (SMoE) models. The proposed method sets the MH-MoE expert intermediate dimension through Eq. (8) and the expert count through Eq. (9), and it reports lower validation perplexities than SMoE and fine-grained SMoE baselines on RedPajama, Wiki, and C4, including with a shared expert and 1-bit BitNet quantization. The paper also contains ablations on the head and merge layers.

Significance. If the parity claim were correct, the paper would offer a simple recipe for converting a standard SMoE architecture into an MH-MoE architecture at equal compute and parameter cost with better perplexity, which would be of practical interest to the MoE community. The paper includes a useful breadth of experiments, including shared experts, 1-bit quantization, and ablations. However, the central FLOPs calculation is inconsistent with the paper's own expert definition, and the reported experimental configurations do not actually satisfy the proposed parity equations. The significance is therefore conditional on a correction that the current manuscript does not provide.

major comments (4)

[Section 2.2, Eqs. (5) and (6)] The complexity model is internally inconsistent. Equation (5) defines each expert with input/output dimension d/h and weight matrices W1 in R^{(d/h) x d_moe} and W2 in R^{d_moe x (d/h)}, so k activated experts cost 2B(d/h)d_moe k scalar multiplications. Equation (6) instead gives the activated-expert leading term as 4B d d_moe k, which is not divided by h and is a factor of two larger than a two-layer FFN count. Because Eq. (8) is derived by balancing this leading term against the SMoE leading term, the parity recipe is not established.
[Section 2.3, Eq. (8), and Section 3] The experimental configurations do not satisfy the proposed Eq. (8). For the SMoE baseline with d=768, d_moe=2048, and k=1, Eq. (8) gives d_mhmoe = 2048 - 768/2 = 1664 for h=2, k=2, and d_mhmoe = 2048 - 768/3 = 1792 for h=3, k=3, while Section 3 reports d_mhmoe = 768 and 512. Using the paper's own Eq. (6) at the reported configurations gives an MH-MoE leading term of 7,077,888B versus 6,291,456B for the SMoE baseline, i.e., about 12.5% higher FLOPs. Parameter parity is also not exact: for h=3, d_mhmoe=512, E=96, Eq. (9) gives 26,345,472 parameters versus 25,165,824 for the SMoE baseline, about 4.7% higher.
[Section 3, SwiGLU versus Eq. (5)] All experiments use SwiGLU feedforward networks, as stated in Section 3, whereas Eq. (5) and Eq. (6) model a two-matrix ReLU FFN. A SwiGLU expert has three weight matrices, so the FLOPs count in Eq. (6) and the parameter count 2(d/h)d_mhmoe used in Eq. (9) do not apply to the evaluated models. The reported comparisons are therefore not controlled tests of the paper's FLOPs and parameter parity claim.
[Section 3.1 and Section 4] Because the parity premise is not established, the lower perplexity values reported for MH-MoE in Tables 1 and 2 cannot be attributed to the multi-head mechanism; they may be due to unequal compute or parameter counts. The central conclusion in Section 4 that MH-MoE "maintains both FLOPs and parameter parity" while improving quality is not supported by the evidence presented.

minor comments (4)

[Section 2.1] The word "ingratiating" should be "integrating".
[Abstract and Section 4] The wording "both outperform both vanilla SMoE models and fine-grained MoE models" contains a doubled "both" and should be rephrased.
[Section 3.3 and Tables 4-5] The ablation table values for MH-MoE (head=2) with head and merge layers are 11.46/9.98/14.89 in Table 4, while Table 1 reports 11.60/10.11/15.11 for what appears to be the same configuration; the discrepancy should be explained or corrected.
[Section 3.3] The sentence "It is crucial to thoroughly investigate their contributions and understand their impact" is missing a closing period.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FLOPs- and parameter-parity recipe follows algebraically from the paper's own complexity model, and the quality claims are validated against external baselines.

full rationale

The central derivation is self-contained. Section 2.2 defines the MH-MoE expert as an FFN over d/h-dimensional sub-tokens (Eq. 5); counting multiplications and additions over h sub-tokens and k activated experts per token yields the activated-expert term in Eq. 6. Equating the leading term of Eq. 6 with the SMoE leading term from Eq. 7 gives Eq. 8, and equating parameter counts gives Eq. 9. These are algebraic manipulations of the paper's own definitions, not fits to the reported perplexities. The experimental section compares MH-MoE against SMoE and fine-grained SMoE under shared code, data, and training setup, so the quality improvements are independent empirical results. The paper does cite prior work by the same group for the original MH-MoE architecture ([WHWW24]) and BitNet ([MWM+24]), but neither citation is used to argue for the parity construction or for the perplexity gains; those are established in this paper. Possible concerns about whether Eq. 6 accurately represents the actual SwiGLU FLOPs are correctness risks, not circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on hand-chosen architectural hyperparameters (h, d_mhmoe, E, top-k) and on a FLOPs model that is inconsistent with the paper's own expert definition. No new physical or theoretical entities are introduced. The parity equations are meant to be exact but contain algebraic and modeling errors, so the design parameters are effectively ad hoc relative to the stated goal.

free parameters (4)

Number of heads h = 2 or 3
Chosen by hand; the experiments show 3 heads is better than 2, but there is no search or theoretical derivation for the optimal value.
MH-MoE expert intermediate dimension d_mhmoe = 768 (h=2), 512 (h=3)
Chosen by hand to hit FLOPs parity; does not match the paper's own Eq. 8 or the corrected formula for d_moe=2048.
Number of MH-MoE experts E = 40 (h=2), 96 (h=3)
Chosen via the parameter-parity equation, but the resulting total expert+head/merge parameters differ from the SMoE baseline by roughly -1.6% and +4.7%.
Top-k gating for MH-MoE = 2 (h=2), 3 (h=3)
Used to compensate for the smaller expert width; no independent derivation is given and parity fails under the corrected formula.

assumptions (3)

domain assumption Eq. 6's activated-expert FLOP count 4Bd d_mhmoe k correctly models the cost of experts whose input dimension is d/h.
Invoked in Section 2.2; it contradicts Eq. 5, which sets expert input to d/h. If instead the correct count includes a factor 1/h, the parity derivation changes.
domain assumption A two-matrix ReLU FFN is an adequate complexity proxy for the SwiGLU FFN used in the experiments.
Section 2.2 derives parity with ReLU FFNs; Section 3 says all models use SwiGLU, which has three matrices and 1.5x the FLOPs/params of the derivation.
standard math Matching the leading FLOP term is sufficient for FLOPs parity.
Section 2.3 drops lower-order terms; for small d (768) and d_mhmoe in the hundreds, the dropped terms are not negligible relative to the claimed differences, so leading-term parity is not exact parity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MH-MoE: Multi-Head Mixture-of-Experts." pith.science (2026). https://pith.science/paper/GD7R4NKT

@misc{pith2026241116205,
  author       = {Pith},
  title        = {Pith review of: MH-MoE: Multi-Head Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GD7R4NKT}},
  note         = {Machine review of arXiv:2411.16205}
}

read the original abstract

Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

[1]

Unified scaling laws for routed language models

Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. arXiv preprint arXiv:2202.01169 , 2022

arXiv 2022
[2]

On the representation collapse of sparse mixture of experts

Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems , 35:34600--34613, 2022

work page 2022
[3]

Redpajama: An open source recipe to reproduce llama training dataset, 2023

Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023

2023
[4]

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR , abs/2401.06066, 2024

arXiv 2024
[5]

Glam: Efficient scaling of language models with mixture-of-experts

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. arXiv preprint arXiv:2112.06905 , 2021

arXiv 2021
[6]

Mixtral of experts

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024

arXiv 2024
[7]

Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition

Kenichi Kumatani, Robert Gmyr, Felipe Cruz Salinas, Linquan Liu, Wei Zuo, Devang Patel, Eric Sun, and Yu Shi. Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition. arXiv preprint arXiv:2112.05820 , 2021

arXiv 2021
[8]

Gshard: Scaling giant models with conditional computation and automatic sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020

arXiv 2006

Show all 17 references

[9]

The era of 1-bit llms: All large language models are in 1.58 bits

Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits. CoRR , abs/2402.17764, 2024

2024 arXiv
[10]

Task-based moe for multitask multilingual machine translation

Hai Pham, Young Jin Kim, Subhabrata Mukherjee, David P Woodruff, Barnabas Poczos, and Hany Hassan Awadalla. Task-based moe for multitask multilingual machine translation. arXiv preprint arXiv:2308.15772 , 2023

2023 arXiv
[11]

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018

2018
[12]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 2019

2019
[13]

Glu variants improve transformer, 2020

Noam Shazeer. Glu variants improve transformer, 2020

2020
[14]

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations , 2017

2017
[15]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-...

2017
[16]

Multi-head mixture-of-experts, 2024

Xun Wu, Shaohan Huang, Wenhui Wang, and Furu Wei. Multi-head mixture-of-experts, 2024

2024
[17]

Sparse moe with language guided routing for multilingual machine translation

Xinyu Zhao, Xuxi Chen, Yu Cheng, and Tianlong Chen. Sparse moe with language guided routing for multilingual machine translation. In Conference on Parsimony and Learning (Recent Spotlight Track) , 2023

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Unified scaling laws for routed language models

Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. arXiv preprint arXiv:2202.01169 , 2022

arXiv 2022

[2] [2]

On the representation collapse of sparse mixture of experts

Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems , 35:34600--34613, 2022

work page 2022

[3] [3]

Redpajama: An open source recipe to reproduce llama training dataset, 2023

Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023

2023

[4] [4]

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR , abs/2401.06066, 2024

arXiv 2024

[5] [5]

Glam: Efficient scaling of language models with mixture-of-experts

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. arXiv preprint arXiv:2112.06905 , 2021

arXiv 2021

[6] [6]

Mixtral of experts

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024

arXiv 2024

[7] [7]

Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition

Kenichi Kumatani, Robert Gmyr, Felipe Cruz Salinas, Linquan Liu, Wei Zuo, Devang Patel, Eric Sun, and Yu Shi. Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition. arXiv preprint arXiv:2112.05820 , 2021

arXiv 2021

[8] [8]

Gshard: Scaling giant models with conditional computation and automatic sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020

arXiv 2006