Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Attention-head routers beat vanilla upcycling by 2.05 points

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Router Upcycling initializes MoE routers from the dense model's attention heads, achieving about 2% higher average zero-shot accuracy than vanilla upcycling on small Qwen 8x0.5B models.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Attention-head initialization for upcycled routers is a reasonable trick worth an ablation, but the 'mixture-of-routers' mechanism is mathematically just a linear gate, and the paper's gains aren't isolated from baseline confounds. the 4 major comments →

arxiv 2509.00679 v1 pith:JRYBNW4S submitted 2025-08-31 cs.CL

Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling

Junfeng Ran , Guangxiang Zhao , Yuhan Wu , Dawei Zhu , Longyun Wu , Yikai Zhao , Tong Yang , Lin Sun
show 2 more authors
Xiangzheng Zhang Sujian Li
This is my paper
classification cs.CL
keywords mixture of expertsupcyclingrouter designattention-based routingexpert specializationrouting diversitysparse MoElanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mixture-of-Experts (MoE) upcycling turns a dense model into a sparse one by copying its feed-forward networks into several experts. The weak point is the router—the component that decides which expert handles each token—because it is typically initialized randomly and uses a simple linear score. This paper proposes Router Upcycling: initialize a set of routers from the query projections of the attention heads in the dense checkpoint, and give each expert a fixed key equal to the average attention key vector. Tokens are then scored against experts with an attention-style dot product across multiple query subspaces, and the scores are combined for top-k routing. The authors report that this beats standard upcycling by 2.05 average points across ten benchmarks and other upcycling router variants by more, at negligible extra cost, and that it produces more diverse token assignments and more specialized experts.

Core claim

The paper's central claim is that MoE upcycling can be improved by upcycling the router together with the experts. Instead of starting from a randomly initialized linear gate, the authors initialize multiple routers from the query projections and average key vectors of the attention heads in the dense checkpoint, pairing heads greedily to reach the desired router count. Each token is then represented by several queries, one per router, and each expert by a fixed key; routing scores are the sum of scaled dot products between queries and keys, followed by softmax over the top-k. The authors claim this is the first router-specific method for upcycling, that it adds negligible overhead (about 1M

What carries the argument

The central mechanism is Mixture-of-Routers attention scoring. An upcycled MoE layer is given m routers, one per concatenated pair of attention heads from the layer immediately before the feed-forward network, and n experts, each with one fixed key vector. Router j projects every token into a low-dimensional query Q_j using the initialized query weight W_j; expert i holds key K_i computed as the average attention key of the dense model. The routing score for expert i is the sum over j of (Q_j · K_i)/sqrt(d'), and a softmax over the top-k of these sums selects the experts. By keeping keys token-independent and letting multiple attention-derived queries score the same token, the mechanism is d

Load-bearing premise

The method assumes that average attention key vectors computed from only 10 iterations of the frozen dense checkpoint give stable, transferable expert features; if those keys are noisy or do not transfer after the feed-forward network is duplicated, the routing gains would vanish.

What would settle it

Run Router Upcycling with the same setup but replace the estimated expert keys with random vectors (or with keys from a different 10-iteration sample) and compare zero-shot averages; if the 2-point gain over Vanilla Upcycling disappears or routing collapses to uniform, the key-estimation step is the cause. Also vary key estimation from 1 to 100 iterations to see whether routing scores and downstream scores stabilize.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Upcycling can be extended from experts to the router: with this method, no part of the sparse model needs to start from a random router.
  • The 2.05-point average gain on the 8x0.5B model persists when the backbone is scaled to 8x1.8B (1.66 points), so the benefit is not tied to a single model size.
  • Because the router adds only about 1M parameters per layer, the method can be dropped into existing upcycling pipelines without meaningful compute or memory cost.
  • Routing-weight analyses show more domain-skewed expert utilization (e.g., a math-specialist expert reaching weight 0.28) and lower pairwise expert output similarity, which the paper ties to expert specialization.
  • The 16-expert configuration shows that adding experts without increasing attention-head dimension hurts both baselines and the proposed method, pointing to head dimensionality as a routing-capacity bottleneck.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method assumes that the frozen dense checkpoint's average attention keys are a good prior for expert identity; a testable extension would be to compare keys estimated from 1, 10, and 100 iterations to see when routing gains stabilize.
  • Because the router transfers attention statistics, it may benefit more from high-quality dense checkpoints; poor checkpoints with noisy attention maps could erode the gain, which the paper does not test.
  • The multi-query formulation is not tied to token-choice top-k routing; it could be combined with expert-choice or dynamic token dropping, though those combinations are outside the paper's scope.
  • A broader principle suggested by the result is that any upcycled component that can be initialized from frozen-checkpoint features may beat random initialization, and routers are only one instance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Router Upcycling, a method for initializing and structuring the routers in Mixture-of-Experts (MoE) upcycling. Instead of a randomly initialized linear router, the authors initialize multiple routers from the attention-head query projections of the frozen dense checkpoint, and initialize expert keys as average attention key vectors computed over 10 training iterations. Routing scores are computed as the sum of attention-like inner products between token queries and expert keys, followed by top-k softmax selection. Experiments on an upcycled Qwen 8x0.5B model (100B tokens) report an average zero-shot score of 45.83 across 10 benchmarks, 2.05 points above the Vanilla Upcycling baseline. The paper also reports scaling experiments (8x1.8B, 16x0.5B), routing-diversity and expert-specialization analyses, and ablations over the number of routers and mixture methods.

Significance. If the reported gains are reliable and attributable to the proposed mechanism, the paper would offer a practical, low-overhead improvement for MoE upcycling and would be the first to explicitly optimize the router in the upcycling setting. The attention-derived initialization idea is simple and plausible, and the reported gains are consistently positive across scales and benchmarks. However, the paper's central framing — that a 'mixture of routers' with an 'attention-like mechanism' provides a more expressive or diverse routing decision surface — is weakened by the mathematical form of the scoring function, which is linear in the token representation before the final softmax. The empirical contribution may still be valuable, but it currently rests on a single run and on comparisons that do not isolate the proposed mechanism. The analysis of routing diversity and expert specialization is a useful step, though it is descriptive rather than causal.

major comments (4)
  1. [Sections 3.2 and 3.3, Eqs. (6)-(8)] The routing score is S_i = x^T (sum_j W_j^T K_i) / sqrt(d'). This is a linear function of x; summing over j before the softmax means the m 'routers' are mathematically equivalent to a single linear router with effective column vectors a_i = sum_j W_j^T K_i / sqrt(d'). Consequently, the claimed 'mixture-of-routers' and 'attention-like' mechanism does not create a richer decision surface than a vanilla linear router; it is a low-rank factorization of a linear map with attention-derived initializations. The reported gains could therefore come entirely from the initialization or from the low-rank structure, not from the multi-query mechanism the paper emphasizes. A controlled ablation is needed: (a) the same architecture with randomly initialized W_j and K_i; (b) a single linear router initialized to the same effective A matrix, so initial decisions are identical. Without such ablations, the
  2. [Section 4.1 and Table 1] All results, including the main 2.05-point average gain and the scaling and ablation tables, are from a single training run with no error bars or multiple seeds. At this scale, run-to-run variance can easily be on the order of a point or more on average zero-shot accuracy, so the headline improvement may be within noise. The paper should report at least 2-3 seeds for the main comparison, or otherwise justify the stability of the reported differences. This is directly load-bearing because every positive claim in Sections 5.1, 5.2, and 6 rests on these point estimates.
  3. [Section 4.3 and Table 1] The Switch Upcycling baseline uses the Switch Transformer top-1 router, as stated in Section 4.1 ('we do not drop any tokens during training except for the Switch Transformer'), while Router Upcycling uses top-2 selection. The paper does not state whether the auxiliary loss (0.02) and router z-loss (0.001) are applied to all baselines or only to Router Upcycling. These differences alone could explain part of the 2.05-point gap. The baselines should be compared under the same top-k, the same auxiliary-loss settings, and the same token-dropping policy, or the paper should explicitly control for these factors.
  4. [Section 3.2 and Section 4.1] The expert keys are estimated as average attention keys over only 10 iterations of the frozen dense model (batch size 1024, sequence length 4096). This is a small fraction of the training data and the paper provides no sensitivity analysis for this number, nor for the greedy cosine-similarity concatenation procedure. The method's entire routing behavior depends on these keys; if they are noisy or unstable, the reported gains may be specific to the chosen estimation schedule. An ablation varying the number of key-estimation iterations (e.g., 1, 10, 100, 1000) and a comparison against random concatenation is necessary to support the claim that attention-derived initialization is the cause of improvement.
minor comments (5)
  1. [Abstract and Section 5.1] The paper says 'over 2%' improvement, but the reported difference is 2.05 absolute points (45.83 vs 43.78), which is about 4.7% relative. Please use 'points' or 'relative %' consistently to avoid ambiguity.
  2. [Section 1, Figure 1] Figure 1 is cited as 'preliminary experiments' but no details are given about the setup, the dense checkpoint, the number of training tokens, or the baselines. Add a short description or a reference to Section 4.
  3. [Section 6.1, Table 3] The claim 'we have conducted all possible ablation studies on the numbers' is too strong and is not accompanied by error bars. Also, when m > n and each expert holds multiple keys, the selection rule ('selected based on the top-k score of any of its keys') is underspecified; clarify how scores are aggregated for an expert with multiple keys.
  4. [Section 5.2] The drop in performance for 16x0.5B is attributed to 'limited head dimensions that constrain routing capacity,' but this is a post-hoc explanation not supported by the presented experiments. A controlled study that varies head dimension while keeping expert count fixed would be needed.
  5. [References and notation] Some references have inconsistent formatting (e.g., [23] contains a raw URL in the author field), and the notation for d' (router dimension) is introduced only after Eq. (5) without an explicit definition before Eq. (6). Please polish these presentation issues.

Circularity Check

1 steps flagged

Mixture-of-routers scoring reduces to a linear gating map; benchmark results remain external evidence.

specific steps
  1. renaming known result [Section 3.3, Equations 7–8]
    "To incorporate attention-mapping scores from collaborative routers, we sum over the query dimension for each token: Si = P j S j i , i = 1, . . . , n, j= 1, . . . , m. (8)"

    By Equations 6–8, Si = Σj (Wj x)ᵀ Ki / √d' = xᵀ (Σj Wjᵀ Ki) / √d'. This is exactly a linear gating score with effective weight vector ai = Σj Wjᵀ Ki / √d'. Summing over routers before any nonlinearity collapses the 'mixture-of-routers' into a single linear map, which is the same functional form as the vanilla linear router the paper compares against. The claimed attention-like multi-query mechanism is thus a reparameterization (low-rank factorization) of a known linear router, renamed as a novel 'mixture-of-routers' method. The paper's architectural novelty claim and its causal explanation of improved routing diversity are therefore not supported by a distinct mechanism; any empirical gains must come from the initialization or factorization, not from the 'mixture' itself.

full rationale

The paper's experimental core is self-contained: Router Upcycling is trained on an external corpus and evaluated on held-out zero-shot benchmarks, so the reported scores are not fitted parameters renamed as predictions. The expert keys are estimated over 10 frozen-model iterations as an initialization, not as a target of prediction, and no equation reduces the final average score to the key-estimation procedure. There is no load-bearing self-citation or imported uniqueness theorem. However, one central derivation step deserves a partial circularity flag: the 'mixture-of-routers attention scoring' in Eqs. 7–8 is, by construction, a single linear function of the token embedding, because the per-router scores are summed before the softmax. This makes the mechanism functionally a low-rank linear router, a known baseline, presented under the new name 'Mixture-of-Routers.' This does not force the empirical benchmark outcomes, but it does undermine the paper's architectural novelty and the claim that the attention-like multi-router design itself drives diversity and gains. Thus the overall circularity is moderate, not absent.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The central claim depends on several hand-picked hyperparameters (router count, dimension, loss weights) and on the unverified assumption that attention statistics from the dense checkpoint transfer to expert routing. The expert keys are a new architectural entity, but they are fitted to the training corpus and lack independent validation.

free parameters (6)
  • m (number of routers) = 8
    Set equal to the number of experts; Table 3 shows m=n=8 gives the best average score.
  • d' (router hidden dimension) = 128
    Concatenating pairs of 64-dimensional attention heads yields this dimension; no sweep over d' is reported.
  • auxiliary loss weight = 0.02
    Standard practice from BASE Layers [20], not tuned here.
  • router z-loss weight = 0.001
    Standard practice from ST-MoE [34], not tuned here.
  • expert key initialization = average attention keys over 10 iterations, batch size 1024, seq len 4096
    Chosen to estimate stable keys; no sensitivity analysis is provided for the number of iterations.
  • greedy concatenation pairing = pairs selected by highest cosine similarity
    Paper states random concatenation degrades performance by ~0.5%, so the pairing scheme is a hand-chosen component of the method.
axioms (4)
  • domain assumption MoE upcycling initialized by duplicating FFN weights is a valid starting point.
    Section 2.2 cites Komatsuzaki et al. [18] and the paper builds on this premise without questioning it.
  • domain assumption Attention query-key inner products are a useful routing signal for expert assignment.
    Section 3.3 uses attention scoring for routing; no theoretical justification is given for why this should transfer to expert selection.
  • ad hoc to paper Average attention keys computed on a small sample of the training corpus provide stable expert keys.
    Section 4.1 describes the 10-iteration estimation; the choice is not justified by sensitivity analysis.
  • ad hoc to paper The greedy cosine-similarity concatenation of attention heads improves routing expressivity.
    Section 3.2 says random concatenation degrades by ~0.5%, but no alternative heuristics are tested, so the greedy choice is ad hoc.
invented entities (2)
  • Expert key embeddings K_i no independent evidence
    purpose: Provide stable per-expert feature vectors used as keys in the attention-like router.
    These are new per-expert parameters initialized from the dense model's attention keys; no external evidence of their validity beyond the paper's own experiments.
  • Mixture-of-Routers mechanism no independent evidence
    purpose: Combines scores from multiple attention-like routers to route tokens.
    A new architecture component whose benefit is only demonstrated in this paper's specific setup.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling." pith.science (2026). https://pith.science/paper/JRYBNW4S

@misc{pith2026250900679,
  author       = {Pith},
  title        = {Pith review of: Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JRYBNW4S}},
  note         = {Machine review of arXiv:2509.00679}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The Mixture-of-Experts (MoE) models have gained significant attention in deep learning due to their dynamic resource allocation and superior performance across diverse tasks. However, efficiently training these models remains challenging. The MoE upcycling technique has been proposed to reuse and improve existing model components, thereby minimizing training overhead. Despite this, simple routers, such as linear routers, often struggle with complex routing tasks within MoE upcycling. In response, we propose a novel routing technique called Router Upcycling to enhance the performance of MoE upcycling models. Our approach initializes multiple routers from the attention heads of preceding attention layers during upcycling. These routers collaboratively assign tokens to specialized experts in an attention-like manner. Each token is processed into diverse queries and aligned with the experts' features (serving as keys). Experimental results demonstrate that our method achieves state-of-the-art (SOTA) performance, outperforming other upcycling baselines.

Figures

Figures reproduced from arXiv: 2509.00679 by Dawei Zhu, Guangxiang Zhao, Junfeng Ran, Lin Sun, Longyun Wu, Sujian Li, Tong Yang, Xiangzheng Zhang, Yikai Zhao, Yuhan Wu.

Figure 1
Figure 1. Figure 1: Performance Comparison between vanilla MoE from scratch and various upcycling models with different routers on several benchmarks in Section 4.2. ∗ This work was conducted during an internship at Qiyuan Tech. ∗∗ Corresponding Authors. Email: {sunlin1, zhangxiangzheng}@360.cn, lisujian@pku.edu.cn Mixture-of-Experts (MoE) models dynamically allocate computa￾tional resources, achieving strong performance acro… view at source ↗
Figure 2
Figure 2. Figure 2: Our proposed Router Upcycling framework. (a) Initializing routers with the Attention modules in a dense model, where WQ and WK are the query and key projections, and the "snowflakes" denote that parameters are frozen. (b) Computing routing scores S between token queries Q and expert keys K using Attention scoring. computed by the attention heads in the preceding layer. These expert features, serving as key… view at source ↗
Figure 3
Figure 3. Figure 3: Average accuracy comparison during training. 4.1 Experimental Setup We conducted our main experiments on an upcycled Qwen 8x0.5B model [23] with approximately 2.1B total parameters and around 0.8B activated parameters. Our architecture maintains the Qwen models’ structure and aligns with Mixtral’s established practice (8 experts/layer). These hyperparameters were set using standard prac￾tices from prior wo… view at source ↗
Figure 4
Figure 4. Figure 4: LM loss comparison during training. through the ablation studies presented in Section 6.1 and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Figure 3 shows that Router Upcycling consistently outper [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average routing weights of Vanilla Upcycling and Router Upcycling in the last layer (layer 23) of the 8x0.5B MoE models. how attention heads are paired and concatenated in the base model. Two rules are applied to form proper models when the number of routers m and keys keys changes compared to a fixed number of experts n: 1. If the router number m < n, attention heads are duplicated n/m times before concat… view at source ↗
Figure 6
Figure 6. Figure 6: Layer-wise average output cosine similarity of 8x0.5B MoE models’ experts [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts

    cs.LG 2026-01 unverdicted novelty 6.0

    L2R improves MoE performance by routing in a low-rank space with Lipschitz-controlled saturated inner-product scoring and multi-anchor mechanisms.

Reference graph

Works this paper leans on

38 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Z. Chi, L. Dong, S. Huang, D. Dai, S. Ma, B. Patra, S. Singhal, P. Bajaj, X. Song, X.-L. Mao, H. Huang, and F. Wei. On the Representation Collapse of Sparse Mixture of Experts, Oct. 2022

    work page 2022

  2. [2]

    Clark, K

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044

    Pith/arXiv arXiv 2019

  3. [3]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  4. [4]

    D. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Sta- bleMoE: Stable Routing Strategy for Mixture of Experts, Apr. 2022

    work page 2022

  5. [5]

    D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wu, Z. Xie, Y . K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, Jan. 2024

    work page 2024

  6. [6]

    Fedus, B

    W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, June 2022

    work page 2022

  7. [7]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027

    Pith/arXiv arXiv 2020

  8. [8]

    A. S. Gordon, Z. Kozareva, and M. Roemmele. Choice of plausible al- ternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical F ormalizations of Commonsense Reason- ing, 2011. URL https://api.semanticscholar.org/CorpusID:434646

    work page 2011

  9. [9]

    Gupta, B

    K. Gupta, B. Thérien, A. Ibrahim, M. L. Richter, Q. Anthony, E. Belilovsky, I. Rish, and T. Lesort. Continual pre-training of large language models: How to (re)warm your model?, 2023. URL https: //arxiv.org/abs/2308.04014

    Pith/arXiv arXiv 2023

  10. [10]

    E. He, A. Khattar, R. Prenger, V . Korthikanti, Z. Yan, T. Liu, S. Fan, A. Aithal, M. Shoeybi, and B. Catanzaro. Upcycling Large Language Models into Mixture of Experts, Oct. 2024

    work page 2024

  11. [11]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus), 2023. URL https://arxiv.org/abs/1606.08415

    Pith/arXiv arXiv 2023

  12. [12]

    Huang, Z

    Q. Huang, Z. An, N. Zhuang, M. Tao, C. Zhang, Y . Jin, K. Xu, K. Xu, L. Chen, S. Huang, and Y . Feng. Harder Tasks Need More Experts: Dynamic Routing in MoE Models, Mar. 2024

    work page 2024

  13. [13]

    Huang, X

    S. Huang, X. Wu, S. Ma, and F. Wei. MH-MoE: Multi-Head Mixture- of-Experts, Nov. 2024

    work page 2024

  14. [14]

    R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3:79–87, 1991. URL https://api.semanticscholar.org/CorpusID:572361

    work page 1991

  15. [15]

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of Experts, Jan. 2024

    work page 2024

  16. [16]

    Joshi, E

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,

    work page

  17. [17]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization,

    work page

  18. [18]

    Komatsuzaki, J

    A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y . Tay, M. Dehghani, and N. Houlsby. Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints, Feb. 2023

    work page 2023

  19. [19]

    URL https://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv

  20. [20]

    Lewis, S

    M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer. BASE Layers: Simplifying Training of Large, Sparse Models, Mar. 2021

    work page 2021

  21. [21]

    Kwiatkowski, J

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Al- berti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answer- ing research. Transactions of the Association for Computational Lin- guistics, 7:452–...

    doi:10.1162/tacl_a_00276 2019

  22. [22]

    Mihaylov, P

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering,

    work page

  23. [23]

    J. Li, Z. Sun, X. He, L. Zeng, Y . Lin, E. Li, B. Zheng, R. Zhao, and X. Chen. LocMoE: A Low-Overhead MoE for Large Language Model Training, May 2024

    work page 2024

  24. [24]

    Roller, S

    S. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash Layers For Large Sparse Models, July 2021

    work page 2021

  25. [25]

    Sakaguchi, R

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: //arxiv.org/abs/1907.10641

    Pith/arXiv arXiv 2019

  26. [26]

    Introducing qwen1.5, February 2024

    Qwen. Introducing qwen1.5, February 2024. URL https://qwenlm.git hub.io/blog/qwen1.5/

    work page 2024

  27. [27]

    Shoeybi, M

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053

    Pith/arXiv arXiv 2020

  28. [28]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023

    2023

  29. [29]

    Shazeer, A

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Jan. 2017

    work page 2017

  30. [30]

    X. Wu, S. Huang, W. Wang, and F. Wei. Multi-Head Mixture-of- Experts, Apr. 2024

    work page 2024

  31. [31]

    Zellers, A

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.or g/abs/1905.07830

    Pith/arXiv arXiv 2019

  32. [32]

    Weber, D

    M. Weber, D. Fu, Q. Anthony, Y . Oren, S. Adams, A. Alexandrov, X. Lyu, H. Nguyen, X. Yao, V . Adams, B. Athiwaratkun, R. Chalamala, K. Chen, M. Ryabinin, T. Dao, P. Liang, C. Ré, I. Rish, and C. Zhang. Redpajama: an open dataset for training large language models, 2024. URL https://arxiv.org/abs/2411.12372

    Pith/arXiv arXiv 2024

  33. [33]

    Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon. Mixture-of-Experts with Expert Choice Routing, Oct. 2022

    work page 2022

  34. [34]

    B. Zoph, I. Bello, S. Kumar, N. Du, Y . Huang, J. Dean, N. Shazeer, and W. Fedus. ST-MoE: Designing Stable and Transferable Sparse Expert Models, Apr. 2022

    work page 2022

  35. [35]

    Z. Zeng, Y . Miao, H. Gao, H. Zhang, and Z. Deng. AdaMoE: Token- Adaptive Routing with Null Experts for Mixture-of-Experts Language Models, Oct. 2024

    work page 2024

  36. [38]

    S. Zuo, X. Liu, J. Jiao, Y . J. Kim, H. Hassan, R. Zhang, T. Zhao, and J. Gao. Taming Sparsely Activated Transformer with Stochastic Ex- perts, Feb. 2022

    work page 2022

  37. [2017]

    URL https://arxiv.org/abs/1705.03551

    Pith/arXiv arXiv

  38. [2018]

    URL https://arxiv.org/abs/1809.02789

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.