Multi-level context Modeling for consistent expert selection in Mixture-of-Experts

Naifan Zhang; Shuhan Huang; Wai Kin Victor Chan; Yang Li; Yuanbo Tang

REVIEW 4 major objections 6 minor 36 references

Multi-level context Modeling for consistent expert selection in Mixture-of-Experts

T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that MoE expert selection is hampered by 'context incompleteness' and that fusing cross-layer and local token context before routing restores consistency and beats strong baselines.

desk verdict Plausible context-aware routing architecture, but the headline C4 numbers cannot be trusted as printed until baselines get matched training budgets and the swapped table entries are corrected. read the letter →

arxiv 2607.16427 v1 pith:25G53OV5 submitted 2026-07-17 cs.CL

Shuhan Huang , Naifan Zhang , Yuanbo Tang , Yang Li , Wai Kin Victor Chan This is my paper

classification cs.CL

keywords Mixture-of-Expertsexpertroutingcontextfusioncross-layerattentiontokensimilarityconsistencylanguagemodelingsparsegating

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the bottleneck in Mixture-of-Experts routing is not the selection mechanism itself but the impoverished representations the router sees. It proposes MCF-MOE, which enriches the router input with two complementary signals: a global cross-layer context aggregated from previous layers' hidden states, and a local token-similarity context gathered from a windowed neighborhood. The claim is that this fused representation yields more stable, semantically coherent expert assignments, which in turn improves language modeling perplexity/bits-per-character and downstream GLUE accuracy across several strong MoE baselines. A sympathetic reader would care because if true, it isolates a relatively cheap, architecture-agnostic fix for MoE instability that could apply to large-scale LLM pretraining.

What carries the argument

The load-bearing object is the MCF-MOE gating module, which replaces the standard single-layer router input with a fused multi-level context vector. It has two branches: a global cross-layer attention over a cache of the last k layer outputs (with layer embeddings and a causal mask) and a local similarity-aware fusion that picks the top-k most similar tokens in a radius-r window and aggregates their hidden states with softmax weights. A cross-attention step combines the two branches, and the result is projected by W_expert to produce top-k routing logits. This two-branch construction is what carries the argument: the paper attributes the consistency gains to this explicit context enrichment,

What would settle it

A controlled replication that trains SMOE-Dropout, HyperRouter, RMOE, and MCF-MOE on C4 with identical data order, tokenizer, compute budget, and convergence criterion; if the bits-per-character gap shrinks to the range seen on Enwiki8 (about 0.1-0.2), the paper's central claim of a large routing-induced gain is not supported.

Watch

Extended reading notes

Core claim

The central discovery claim is that routing decisions in MoE models suffer from context incompleteness: because each token's gating logits are computed from a single layer's local representation, assignments are noisy and inconsistent across layers, and semantically related tokens get scattered among experts. MCF-MOE addresses this by (1) cross-attending the current layer's query to a cache of previous layer outputs (with a causal mask) to inject global semantic consistency, and (2) computing a local similarity-aware aggregation over a windowed top-k of neighboring tokens to inject fine-grained token-level cues. These two contexts are fused via cross-attention and then fed to the expert proj

Load-bearing premise

The headline gains rest on the premise that all baselines were trained to comparable convergence with identical compute, data, and hyperparameters; the paper reports a roughly 3x bits-per-character gap on C4 that routing changes alone cannot explain, and it does not specify per-baseline compute budgets.

Editorial extensions

If this is right

If router inputs are context-complete, MoE routing becomes more consistent across layers and more semantically coherent, reducing representation collapse and improving expert specialization.
The context-fusion fix transfers to a 16B-parameter decoder-only backbone, suggesting it is not specific to small Transformer-XL experiments.
The fused router is robust to different top-k activation counts and hyperparameter settings, maintaining performance where baseline routers degrade.
Improved routing consistency translates to downstream gains across six GLUE tasks, not just language modeling.
The overhead is modest: peak inference memory stays within about 9% of the most efficient baseline and inference latency scales favorably at longer sequence lengths.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The causal mask in the cross-layer attention may be unnecessarily conservative: because the cache stores hidden states from earlier layers at the same positions, a model-level causal constraint might allow attending to same-position tokens in later layers without leakage, potentially strengthening the global signal.
The paper's principle suggests a broader design rule: any router that consumes only per-token, per-layer features is information-starved; an extension is to apply similar context fusion to expert output aggregation, not just gate selection.
The near-zero NMI lift for RMOE hints that GRU-based memory routers may encode sequence-order rather than semantic-cluster information; a testable extension is to compare MCF-MOE's fusion against a GRU augmented with a semantic-clustering objective.
The magnitude of the C4 gain (about 2 to 4 bits per character) is far larger than the routing change alone can plausibly explain; an editor's bet is that matched training convergence across baselines would shrink this gap, making a fair head-to-head the natural next experiment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the core idea is legitimate — build the routing gate from cross-layer cached representations plus local top-k token similarity rather than from the current layer's hidden state alone. That combination is not in the cited baselines, and the ablations show both branches contribute. If reproduced at small scale, it is a reasonable incremental contribution to MoE routing.

The paper is less convincing where it claims large gains. The C4/DeepSeek-MoE result in Table 1 says MCF-MOE hits 1.099 bpc while SMOE-Dropout gets 3.058 and RMOE 4.283. A routing change should not move bpc by 2–3 units unless the baselines are undertrained or the evaluation differs. Section 4.1.2 only says “most experiments” run on four A800s; there is no per-baseline compute, token count, convergence criterion, or explicit statement that the 80k-step budget in Appendix B.1 applies to the C4 runs. Without that, the gap is not attributable to routing.

The internal numeric pipeline also has problems. Table 1 and Table 8 swap Enwiki8/WikiText-103 results for SMOE-Dropout and HyperRouter (compare 1.141 vs 1.262 and 26.405 vs 39.557 across the two tables). WNLI reports 20.31 for RMOE, below chance on a binary task, which suggests an evaluation bug. There are no error bars on the main table, so fine-tuning differences are hard to judge; CoLA is described in the appendix as MCC, but the table presents it in the same accuracy scale as the other tasks. The analysis sections (NMI, Gini, expert utilization) support the mechanism qualitatively, but they are diagnostics on the same models rather than independent confirmation.

What should be credited: the architecture is simple to state, the small-scale Transformer-XL results are plausible, and the ablation story is coherent — removing global or local context hurts, cross-attention beats concat or gating, and hyperparameter sensitivity is moderate. That is the useful core.

Who is this for: people working on sparse MoE routing who want another context-aware gating idea. It deserves a serious referee, but the C4 claim needs matched-baseline reruns and the table inconsistencies need fixing before the headline results can be believed.

Referee Report

4 major / 6 minor

Summary. The paper proposes MCF-MoE, a sparse routing framework that replaces the standard router input with a fused representation. A global branch computes cross-attention between the current layer's hidden states and cached outputs from previous layers (with a causal mask), and a local branch selects the top-k tokens in a fixed-radius window by dot-product similarity and aggregates them; the two are combined by cross-attention before the expert logits. The paper reports language-modeling results on Enwiki8, WikiText-103, and C4, GLUE fine-tuning, and analyses of routing consistency (NMI lift, expert entropy, co-occurrence, Gini). The central claim is that enriching routing context with global and local signals produces more stable, semantically coherent expert assignment and consistent downstream gains.

Significance. If the experimental results are trustworthy, the contribution is a simple, generalizable router that could be plugged into existing MoE Transformers, and the routing-consistency analysis is a useful addition. The permutation-based NMI lift and the component ablations are good practices. However, the current support is weakened by the anomalously large C4 gap, unresolvable inconsistencies between Table 1 and Table 8, and below-chance WNLI numbers, so the central claim of consistent improvements is not yet established.

major comments (4)

[Table 1 (C4 rows), §4.1.2, Appendix B.1] The C4/DeepSeek-MoE columns are the strongest quantitative evidence for scalability, but the reported gaps are implausible for a routing change alone. MCF-MoE is listed at 1.099 bpc, SMOE-Dropout at 3.058 bpc, RMOE at 4.283 bpc, and SMOE at 5.294 bpc; a top-2-of-16 routing modification should not shift bpc by 2-4 units unless baselines are severely undertrained or the evaluation pipeline differs. Section 4.1.2 gives GPU count but no compute budget, token count, optimizer schedule, or convergence criterion for the C4 experiments; Appendix B.1's 80k steps is explicitly for the Transformer-XL setting. Please run all routers with matched data, tokenizer, training steps/compute, report convergence curves, and give eval details. If the C4 gap shrinks, the large-scale claim in the abstract and §4.2 must be revised.
[Table 1 vs Table 8] There is an internal swap between the two tables for Enwiki8/WikiText-103. Table 1 reports SMOE-Dropout as (1.262 bpc, 39.557 ppl) and HyperRouter as (1.141 bpc, 27.128 ppl); Table 8 assigns the (1.262, 39.557) pair to HyperRouter in the K=2 row and gives SMOE-Dropout values of 1.141/27.128 in other rows. Since Table 1 is the centerpiece of the empirical contribution, this inconsistency means the reported numbers cannot be taken as reliable until the experimental log and eval script are checked and the tables corrected.
[Table 1 fine-tuning rows] Several WNLI accuracies are below the 50% random baseline for binary classification: RMOE 20.31, Dense 23.44, SMOE 32.81. The paper uses these numbers in the average that supports the claim that MCF-MoE outperforms all baselines. This is implausible under the standard GLUE WNLI protocol and suggests an evaluation/label misalignment for at least some of the listed models. Please report exact evaluation code, label mapping, and test split used, and re-evaluate all methods; if a method genuinely scores below chance, that should be explained.
[§4.5.1, Eq. (17)-(18); Appendix G] The consistency analyses used to support the mechanism are not fully defined. Section 4.5.1 says token occurrences are grouped using a 'frozen external encoder' but does not identify the encoder, clustering algorithm, number of clusters, or which layer's expert assignments are used; Figure 4's error bars are over random seeds, but no seed count is given. Appendix G reports 'Gini consistency' as a single score without defining how routing decisions are converted into a Gini coefficient. Because these metrics are central to the 'routing consistency' claim, please provide full formal definitions and code/configurations.

minor comments (6)

[§4.3.1, §4.3.2] The two ablation subsections have the same title; the second should be renamed, e.g., 'Ablation of fusion strategy, depth, and history window'.
[Appendix B.2] The text says fine-tuning uses four GLUE datasets (SST-2, QQP, QNLI, RTE), but Table 1 includes CoLA and WNLI. List all six datasets.
[Table 3] The 'Val Test' columns have no metric label; presumably bpc, but this should be stated explicitly.
[Appendix A.2 / Table 1] Appendix A.2 says CoLA is evaluated with MCC and QQP with accuracy/F1, but Table 1's heading says accuracy. Clarify which metric is reported for each task.
[Figure 3 / Table 4] Latency and memory comparisons lack experimental context: hardware configuration, batch size, sequence-length packing, and profiling methodology. Specify them.
[Table 8] The caption says 'varying top-K', but the checkpoint selection and evaluation split are unspecified; report whether each K setting was trained from scratch or derived from the same checkpoint.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the paper's claims are empirical comparisons, not fitted predictions.

full rationale

The paper makes no first-principles derivation; its claims are empirical comparisons between MCF-MOE and baselines. The routing equations (Eqs. 1-16) define the proposed method rather than deriving a predicted result from fitted inputs. No parameter is fit to a subset of data and then reported as a prediction of a closely related quantity; the ablations in Table 2 compare entire trained models, and the NMI/Gini analyses in Section 4.5 and Appendix G describe routing behavior post hoc. No load-bearing self-citation chain is present: the motivating citations (Arnold et al. 2024; Li et al. 2024) are external, and the paper's contribution is a new architecture, not a uniqueness theorem. Appendix J explicitly acknowledges limitations (robust fusion, hyperparameter sensitivity, computational overhead), and the unverified C4 training-budget match (Table 1; Section 4.1.2) is an evaluation/correctness concern, not a circularity. Under the definitional-fit standard, no output equals an input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces three hyperparameters (history window k, local radius r, local top-k) tuned on validation data, and relies on assumptions that cross-layer cached outputs and locally similar tokens provide useful routing signals. No new physical entities are introduced.

free parameters (3)

history window k = 3
Number of historical layers in global cross-layer fusion; chosen by ablation on Enwiki8 val (Table 3), k=3 best.
local radius r = 16
Window size for local similarity; chosen by sensitivity analysis (Table 9), r=16 with top-k=8 best.
local top-k neighbors = 8
Number of similar tokens aggregated in local fusion; chosen by sensitivity analysis (Table 9).

assumptions (4)

domain assumption Cross-layer attention with a causal mask preserves autoregressive consistency
Equation (5) with mask (3); necessary for valid language modeling, but not proven beyond construction.
domain assumption Tokens that are similar within a local window should share experts
This is the design premise of §3.2; semantic coherence of routing is assumed beneficial.
domain assumption Cached historical layer outputs provide useful routing context
Global cross-layer fusion assumes earlier layer representations carry complementary routing-relevant semantics (§3.1).
standard math Softmax top-k expert selection is a standard MoE primitive
Used in Eq. (16), following SMOE convention (Shazeer et al. 2017).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-level context Modeling for consistent expert selection in Mixture-of-Experts." pith.science (2026). https://pith.science/paper/25G53OV5

@misc{pith2026260716427,
  author       = {Pith},
  title        = {Pith review of: Multi-level context Modeling for consistent expert selection in Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25G53OV5}},
  note         = {Machine review of arXiv:2607.16427}
}

read the original abstract

Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts. However, existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers. In this work, we revisit expert selection from a representation perspective and identify context incompleteness as a key bottleneck limiting effective expert specialization. To address this issue, we propose Multi-level Context Fusion MOE (MCF-MOE), a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions, enabling more informative and consistent expert selection. Experiments on language modeling and understanding benchmarks demonstrate that MCF-MOE consistently improves routing consistency and downstream performance over strong MoE baselines, highlighting the importance of contextual completeness in expert routing. The code is available at https://anonymous.4open.science/r/MCFMOE.

Figures

Figures reproduced from arXiv: 2607.16427 by the authors.

**Figure 2.** Overall workflow of MCF-MOE. Left: a Transformer layer with the MCF-MOE gate. Middle: the Global–local Context Fusion module, containing two complementary branches — (1) Global cross-layer context fusion, which cross-attends the current input embedding (query) to cached historical-layer outputs; and (2) Local similarity-aware fusion, which computes token-level similarities within a local window and retrieves the top… view at source ↗

**Figure 3.** Comparison of model’s inference latency across different sequence length. Inference Memory [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Related-token routing specialization measured [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Layer-wise comparison of expert usage en [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 7.** Figure 7: Comparison of expert utilization across all layers in four routing methods. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Gini consistency scores of MCF-MOE and baseline models, evaluates the stability and consistency of expert routing across different model variants. Higher is better. H Model scalability analysis To assess whether the benefits of MCF-MOE persist as model size increases, …

**Figure 9.** Figure 9: Performance of different gating mechanisms [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 5 canonical work pages

[1]

Scaling Laws Across Model Architectures:

Siqi Wang and Zhengyu Chen and Bei Li and Keqing He and Min Zhang and Jingang Wang , editor =. Scaling Laws Across Model Architectures:. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.319 , timestamp =

doi:10.18653/v1/2024.emnlp-main.319 2024
[2]

arXiv preprint arXiv:2503.20750 , year=

Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework , author=. arXiv preprint arXiv:2503.20750 , year=

arXiv
[3]

2025 , url =

Weilin Cai and Juyong Jiang and Fan Wang and Jing Tang and Sunghun Kim and Jiayi Huang , title =. 2025 , url =. doi:10.1109/TKDE.2025.3554028 , timestamp =

arXiv 2025
[4]

Advances in Neural Information Processing Systems , volume=

Scaling vision with sparse mixture of experts , author=. Advances in Neural Information Processing Systems , volume=
[5]

Advances in Neural Information Processing Systems , volume=

Moeut: Mixture-of-experts universal transformers , author=. Advances in Neural Information Processing Systems , volume=
[6]

IEEE Transactions on Parallel and Distributed Systems , volume=

Mpmoe: Memory efficient moe for pre-trained models with adaptive pipeline parallelism , author=. IEEE Transactions on Parallel and Distributed Systems , volume=. 2024 , publisher=

2024
[7]

JOURNAL OF IEEE TRANSACTIONS ON ARTIFICIAL INTELLIGENCE , year=

The evolution of moe: A survey from basics to breakthroughs , author=. JOURNAL OF IEEE TRANSACTIONS ON ARTIFICIAL INTELLIGENCE , year=
[8]

CoRR , volume =

Xianzhi Du and Tom Gunter and Xiang Kong and Mark Lee and Zirui Wang and Aonan Zhang and Nan Du and Ruoming Pang , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2405.15052 , eprinttype =. 2405.15052 , timestamp =

arXiv doi:10.48550/arxiv.2405.15052 2024

Show all 36 references

[9]

Scaling Vision with Sparse Mixture of Experts , booktitle =

Carlos Riquelme and Joan Puigcerver and Basil Mustafa and Maxim Neumann and Rodolphe Jenatton and Andr. Scaling Vision with Sparse Mixture of Experts , booktitle =. 2021 , url =

2021
[10]

The Thirteenth International Conference on Learning Representations,

Tongtian Yue and Longteng Guo and Jie Cheng and Xuange Gao and Hua Huang and Jing Liu , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

2025
[11]

Xinglin Pan and Wenxiang Lin and Lin Zhang and Shaohuai Shi and Zhenheng Tang and Rui Wang and Bo Li and Xiaowen Chu , editor =. FSMoE:. Proceedings of the 30th. 2025 , url =. doi:10.1145/3669940.3707272 , timestamp =

2025
[12]

Le and Geoffrey E

Noam Shazeer and Azalia Mirhoseini and Krzysztof Maziarz and Andy Davis and Quoc V. Le and Geoffrey E. Hinton and Jeff Dean , title =. 5th International Conference on Learning Representations,. 2017 , url =

2017
[13]

William Fedus and Barret Zoph and Noam Shazeer , title =. J. Mach. Learn. Res. , volume =. 2022 , url =

2022
[14]

The Eleventh International Conference on Learning Representations,

Tianlong Chen and Zhenyu Zhang and Ajay Kumar Jaiswal and Shiwei Liu and Zhangyang Wang , title =. The Eleventh International Conference on Learning Representations,. 2023 , url =

2023
[15]

On the Representation Collapse of Sparse Mixture of Experts , booktitle =

Zewen Chi and Li Dong and Shaohan Huang and Damai Dai and Shuming Ma and Barun Patra and Saksham Singhal and Payal Bajaj and Xia Song and Xian. On the Representation Collapse of Sparse Mixture of Experts , booktitle =. 2022 , url =

2022
[16]

SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse , booktitle =

Giang Do and Hung Le and Truyen Tran , editor =. SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse , booktitle =. 2025 , url =. doi:10.18653/V1/2025.FINDINGS-NAACL.107 , timestamp =

2025 doi
[17]

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts , booktitle =

Truong Do and Le Khiem and Quang Pham and TrungTin Nguyen and Thanh. HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts , booktitle =. 2023 , url =. doi:10.18653/V1/2023.EMNLP-MAIN.351 , timestamp =

2023 doi
[18]

The Thirteenth International Conference on Learning Representations,

Zihan Qiu and Zeyu Huang and Shuang Cheng and Yizhi Zhou and Zili Wang and Ivan Titov and Jie Fu , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

2025
[19]

Oseledets , title =

Daniel Bershatsky and Ivan V. Oseledets , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.04444 , eprinttype =. 2504.04444 , timestamp =

2025 doi arXiv
[20]

CoRR , volume =

Jingcong Liang and Siyuan Wang and Miren Tian and Yitong Li and Duyu Tang and Zhongyu Wei , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.16056 , eprinttype =. 2505.16056 , timestamp =

2025 doi
[21]

CoRR , volume =

Stefan Arnold and Marian Fietta and Dilara Yesilbas , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2409.14107 , eprinttype =. 2409.14107 , timestamp =

2024 doi arXiv
[22]

arXiv preprint arXiv:2406.00023 , year=

Expert-token resonance: Redefining moe routing through affinity-driven active selection , author=. arXiv preprint arXiv:2406.00023 , year=
[23]

StableMoE: Stable Routing Strategy for Mixture of Experts , booktitle =

Damai Dai and Li Dong and Shuming Ma and Bo Zheng and Zhifang Sui and Baobao Chang and Furu Wei , editor =. StableMoE: Stable Routing Strategy for Mixture of Experts , booktitle =. 2022 , url =. doi:10.18653/V1/2022.ACL-LONG.489 , timestamp =

2022 doi
[24]

THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation , booktitle =

Yunlong Liang and Fandong Meng and Jie Zhou , editor =. THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation , booktitle =. 2025 , url =

2025
[25]

CoRR , volume =

Nabil Omi and Siddhartha Sen and Ali Farhadi , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.14038 , eprinttype =. 2506.14038 , timestamp =

2025 doi
[26]

Proceedings of the IEEE/CVF international conference on computer vision , pages=

Swin transformer: Hierarchical vision transformer using shifted windows , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
[27]

Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation , booktitle =

Mozhdeh Gheini and Xiang Ren and Jonathan May , editor =. Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation , booktitle =. 2021 , url =. doi:10.18653/V1/2021.EMNLP-MAIN.132 , timestamp =

2021 doi
[28]

Advances in neural information processing systems , volume=

Attention is all you need , author=. Advances in neural information processing systems , volume=
[29]

Carbonell and Quoc Viet Le and Ruslan Salakhutdinov , editor =

Zihang Dai and Zhilin Yang and Yiming Yang and Jaime G. Carbonell and Quoc Viet Le and Ruslan Salakhutdinov , editor =. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context , booktitle =. 2019 , url =. doi:10.18653/V1/P19-1285 , timestamp =

2019 doi
[30]

Large text compression benchmark , author=
[31]

arXiv preprint arXiv:1609.07843 , year=

Pointer sentinel mixture models , author=. arXiv preprint arXiv:1609.07843 , year=

arXiv
[32]

Manning and Andrew Y

Richard Socher and Alex Perelygin and Jean Wu and Jason Chuang and Christopher D. Manning and Andrew Y. Ng and Christopher Potts , title =. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,. 2013 , url =. doi:10.18653/V1/D13-1170 , timestamp =

2013 doi
[33]

arXiv preprint arXiv:1804.07461 , year=

GLUE: A multi-task benchmark and analysis platform for natural language understanding , author=. arXiv preprint arXiv:1804.07461 , year=

arXiv
[34]

CoRR , volume =

Ben Fauber , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2411.07983 , eprinttype =. 2411.07983 , timestamp =

2024 doi arXiv
[35]

MoEC: Mixture of Expert Clusters , booktitle =

Yuan Xie and Shaohan Huang and Tianyu Chen and Furu Wei , editor =. MoEC: Mixture of Expert Clusters , booktitle =. 2023 , url =. doi:10.1609/AAAI.V37I11.26617 , timestamp =

2023 doi
[36]

Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Mode: A mixture-of-experts model with mutual distillation among the experts , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Scaling Laws Across Model Architectures:

Siqi Wang and Zhengyu Chen and Bei Li and Keqing He and Min Zhang and Jingang Wang , editor =. Scaling Laws Across Model Architectures:. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.319 , timestamp =

doi:10.18653/v1/2024.emnlp-main.319 2024

[2] [2]

arXiv preprint arXiv:2503.20750 , year=

Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework , author=. arXiv preprint arXiv:2503.20750 , year=

arXiv

[3] [3]

2025 , url =

Weilin Cai and Juyong Jiang and Fan Wang and Jing Tang and Sunghun Kim and Jiayi Huang , title =. 2025 , url =. doi:10.1109/TKDE.2025.3554028 , timestamp =

arXiv 2025

[4] [4]

Advances in Neural Information Processing Systems , volume=

Scaling vision with sparse mixture of experts , author=. Advances in Neural Information Processing Systems , volume=

[5] [5]

Advances in Neural Information Processing Systems , volume=

Moeut: Mixture-of-experts universal transformers , author=. Advances in Neural Information Processing Systems , volume=

[6] [6]

IEEE Transactions on Parallel and Distributed Systems , volume=

Mpmoe: Memory efficient moe for pre-trained models with adaptive pipeline parallelism , author=. IEEE Transactions on Parallel and Distributed Systems , volume=. 2024 , publisher=

2024

[7] [7]

JOURNAL OF IEEE TRANSACTIONS ON ARTIFICIAL INTELLIGENCE , year=

The evolution of moe: A survey from basics to breakthroughs , author=. JOURNAL OF IEEE TRANSACTIONS ON ARTIFICIAL INTELLIGENCE , year=

[8] [8]

CoRR , volume =

Xianzhi Du and Tom Gunter and Xiang Kong and Mark Lee and Zirui Wang and Aonan Zhang and Nan Du and Ruoming Pang , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2405.15052 , eprinttype =. 2405.15052 , timestamp =

arXiv doi:10.48550/arxiv.2405.15052 2024