Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding a sequence-level clustering stage before token-level expert selection lets Mixture-of-Experts models specialize expert groups and generalize across math, code, knowledge, and reasoning tasks.

desk verdict Solid empirical MoE paper with a plausible dual-stage routing mechanism; the ablations hold up better than the stress-test suggests, but reporting gaps keep the effect size uncertain. read the letter →

arxiv 2509.10513 v1 pith:BUPXXQ7J submitted 2025-09-03 cs.LG

Sugyeong Eo , Jungjun Lee , Chanjun Park , Heuiseok Lim This is my paper · ORCID
classification cs.LG
keywords mixtureofexpertsinstructiontuningexpertspecializationdual-stageroutingk-meansclusteringsequence-levelparameter-efficientfine-tuninglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish why a sparse Mixture-of-Experts model fails to specialize when instruction-tuning data is highly heterogeneous: token-level routing by itself cannot group inputs by task or knowledge requirement. MoCE's fix is a dual-stage router—first a k-means cluster of the whole input embedding picks one expert group, then a token-level top-k gate picks experts inside that group. The paper argues this forces expert groups to specialize on distinct input clusters while keeping the computational cost of ordinary MoE, and reports consistent gains over adapter-based MoE baselines and over models trained on a single domain. The significance, if the claim holds, is that expert specialization and cross-task generalization can be induced by a simple precomputed clustering without increasing active parameters.

What carries the argument

The load-bearing object is the grouped MoE layer with a two-stage router. In the first stage, a fixed encoder embedding model (E5 or Instructor) represents the full input sequence; k-means assigns it to one of $M$ clusters, and a one-to-one mapping activates the corresponding expert group $G_{\alpha}$. In the second stage, a group-specific gating function $R_{\alpha}$ applies the standard top-k sparse gating among the group's experts, producing $y = \sum_i \mathrm{TopK}(R_{\alpha}(x)_i,k)\,A_{G_{\alpha},i}(x)$. In the adapter-based setting used here, each expert is a low-rank adaptation (an up/down projection pair with a residual connection) on a frozen feed-forward network. A variant sums in general experts and a general router, pooling knowledge from all clusters. The machinery forces different expert groups to be exposed only to inputs whose embeddings fall in their cluster, while top-k selection keeps token-level flexibility.

What would settle it

Permute the one-to-one mapping from cluster IDs to expert groups, retrain only the router on the same data, and evaluate on the same benchmarks; if average performance does not fall below the standard cluster mapping, the clustering content is not driving the gains.

Watch

Extended reading notes

Core claim

MoCE's central claim is that a two-level routing hierarchy—sequence-level group allocation followed by token-level expert selection—makes experts specialize by knowledge cluster and generalize to out-of-domain inputs better than token-only MoE. On the paper's experiments, MoCE-E5 averages 37.99 and MoCE-Instructor 37.61 across math, code, knowledge, and reasoning benchmarks, against 35.62 for PESC, 32.44 for BTX (A), and 32.26 for LLaMA-Adapter; gains are largest on GSM8K (41.93 vs 33.21 for PESC) and HumanEval@1 (19.28 vs 16.00). Domain-tuned PESC models score higher than MoCE only within their narrow domain and fall behind elsewhere, which the paper reads as evidence that MoCE balances specialization with generalization. The method transfers to Mistral and Qwen backbones and, with four or seven clusters chosen by an elbow rule, beats publicly released dense and MoE chat models of comparable size.

Load-bearing premise

The first routing stage assumes the clusters inferred from training-sequence embeddings keep their meaning on test inputs; if a new benchmark's examples do not sit near the training cluster centers, the cluster-to-group mapping is effectively arbitrary.

Editorial extensions

If this is right

  • MoCE's dual-stage routing improves average performance over adapter-based MoE baselines on all four task categories without activating more experts per token.
  • Domain-specialized PESC models are matched or beaten by MoCE in their own domains on math and code while MoCE retains general-task performance, suggesting grouping avoids sacrificing cross-domain transfer.
  • The same cluster-group architecture transfers to Mistral and Qwen model families with consistent gains, so the mechanism is not tied to one backbone.
  • Scaling the number of experts per group increases accuracy on code, math, and general benchmarks while keeping the number of activated experts at top-2.
  • Adding a set of general experts alongside the clustered groups lifts average performance, so domain-grouped knowledge and general knowledge combine productively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the clustering encoder is frozen and external, a natural test is to replace it with the MoE model's own hidden states or with trainable centroids; that would isolate how much of the gain comes from the routing signal itself versus the fixed group structure.
  • The architecture could be deployed at lower memory cost by loading only the expert group assigned to the current cluster during inference; the paper does not test this, but the one-to-one cluster-to-group mapping makes it a direct extension.
  • The authors' own stated limits—adapters only, no multilingual or broader-domain evaluation—mean the specialization claim is currently demonstrated for English math, code, knowledge, and reasoning tasks; extending to full FFN MoE layers and other languages would test whether the grouping benefit is generic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Mixture-of-Clustered-Experts (MoCE), a sparse MoE architecture for instruction tuning in which experts are organized into groups. A dual-stage routing mechanism first assigns each input sequence to an expert group via k-means clustering over encoder-based sequence embeddings, then performs standard top-k token-level routing within the selected group. The method is implemented in an adapter-based, upcycled setting and evaluated on LLaMA2-7B, Mistral-7B, and Qwen-7B across mathematics, code, knowledge, and reasoning benchmarks. The authors report consistent improvements over adapter-based baselines, including PESC and domain-specialized PESC models, and provide ablations on routing strategies and expert-count scaling.

Significance. If the reported gains are robust, MoCE offers a simple, practical recipe for improving expert specialization and cross-domain generalization in instruction-tuned MoE models, with evidence across three model families and multiple public benchmarks. The paper's strengths include evaluation on standard benchmarks, ablations of both routing stages, and experiments with different backbone models and embedding models. The main weakness is that the central claim—that hierarchical cluster-based routing, rather than additional parameters or selection artifacts, drives the improvements—is not yet established with the necessary controls and statistical evidence.

major comments (4)
  1. [§5.4, Figure 4] The cluster count is partly selected after inspecting test performance, which blurs selection and validation. The text states that the elbow method identifies four and seven as the optimal counts for Instructor and E5 embeddings, respectively, and Figure 4 then shows that these same counts 'yield the best overall performance' on the evaluation benchmarks. Since the evaluation benchmarks are used to confirm the elbow-chosen count, the claim that the elbow method is validated as a reliable approach is circular. Please report the elbow-selected counts based only on training-set embeddings, and treat any performance-based adjustment as model selection on the test set, with the corresponding caveat about optimistic bias.
  2. [§3.2–3.3, Figure 5] The transfer of cluster centroids from the training mixture to evaluation inputs is not quantitatively validated. The centroids are fit on SlimOrca, Magicoder, and MetaMathQA and then used to assign evaluation sequences, but the only evidence is the qualitative 100-sample visualization in Figure 5. There is no measure of cluster purity, assignment stability, or comparison against random assignment of sequences to expert groups. Since the sequence-level stage is the defining contribution of MoCE, this unvalidated transfer is load-bearing: if assignment on evaluation inputs is effectively arbitrary, the gains in Table 1 could arise from the larger total expert bank rather than from meaningful routing. Please add a quantitative analysis of cluster-transfer quality, such as agreement between train and test embedding geometries, cluster-size balance on evaluation sets, or an ablation that randomizes the cluster assignment while holding the number of experts fixed.
  3. [Table 1 and Table 3] The comparison against PESC is confounded with total expert count. Table 3 shows that increasing the number of experts per cluster from 1 to 4 (i.e., from 4 to 16 total experts) improves the average score from 35.55 to 36.76, and the paper does not report how many experts PESC uses. If PESC uses fewer than 16 experts, the gains attributed to hierarchical routing in Table 1 may partly reflect parameter count. Please report the total expert count for PESC and, if feasible, include a PESC baseline with a matched total number of experts, or otherwise demonstrate that the improvement is not explained by the expert-count scaling shown in Table 3.
  4. [Table 1, Figure 3, Table 4] All reported scores are single-run point estimates without error bars or significance tests. Many of the comparisons involve small margins (for example, MMLU-Pro differences of 0.3–0.9 points in Table 1), so it is unclear whether the improvements are statistically reliable. At minimum, please provide multiple seeds or bootstrap confidence intervals for the main comparisons against PESC and for the ablations in Table 4. This is important for the central claim of consistent superiority.
minor comments (6)
  1. [§3.1, Eq. (1)-(3)] The notation for the gating function is inconsistent: R(x)_i is used in Eq. (1) but TopK(R(x)_i, k) is written with the index as a subscript; please clarify the indexing convention.
  2. [Table 2 caption] The caption says 'Comparative evaluation results with dense and MoE models' but the table only includes dense models of similar size and MoE-based models; it is not clear which rows are dense and which are MoE. Please label the model types explicitly.
  3. [§5.1, first paragraph] The sentence 'MoCE demonstrates superior performance not only over general baselines but also over models individually optimized for specific domains' is stated before the domain-specialized results are shown; consider moving this claim to after the relevant comparison.
  4. [Table 4] The ablation 'w/o Clustering' is not precisely defined. If clustering is disabled, how is the expert group selected? Please specify the alternative assignment used in this ablation.
  5. [Figure 5] The subfigure labels are numbered '(2)' twice and skip '(3)'; please correct the numbering.
  6. [Appendix A] The adapter dimension, number of experts, and top-k are stated, but the load-balancing loss is only mentioned in §4.1; please specify its coefficient or provide the exact loss formulation used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoCE is an empirical architecture study whose central claims rest on held-out benchmark evaluations, not on a derivation from its own inputs.

full rationale

The paper's central claim is that its dual-stage routing mechanism improves expert specialization and generalization. This is supported by held-out evaluations on standard benchmarks (Table 1), ablations (Table 4), and cross-model-family tests (Figure 3). There is no first-principles derivation that could collapse into an input by construction. The k-means centroids are fit on training data only, and evaluation inputs are assigned by nearest centroid; this is a standard inductive procedure, and the paper does not rename a fitted quantity as a prediction. The elbow method for choosing the number of clusters is a hyperparameter-selection step, and its subsequent confirmation via performance curves (Figure 4) is a sensitivity/validation analysis rather than a circular derivation, because the cluster count is not inferred from the test labels used to report final performance. The only weak point is the qualitative 100-sample visualization in Figure 5, which is weak evidence of cluster transfer, but absence of strong validation is a correctness/robustness concern, not circularity. No load-bearing self-citation chain appears: the cited PESC adapter framework, embedding models, and benchmarks are external, and the authors' own contributions are tested against those external baselines. Overall, the derivation chain is self-contained as an empirical study, so no circular steps are identifiable under the stated criteria.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard clustering math plus several domain assumptions about embeddings and evaluation transfer; no code or error bars are provided. The cluster count is a hand/elbow-selected hyperparameter, and the cluster-to-group mapping is an architectural choice.

free parameters (4)
  • Number of clusters / expert groups (M) = 4 (Instructor); 7 (E5)
    Set by elbow method on training embeddings and then checked against test performance in Figure 4; directly controls group granularity.
  • Experts per group (N) = 4
    Fixed at 4 in main experiments; scaling study in Table 3 shows more experts per group improves average score.
  • Top-k token selection = 2
    Chosen after comparing top-1, top-2 and soft merging in Table 4; top-2 and soft merging perform best.
  • Adapter dimension = 64
    Hyperparameter inherited from adapter setup; no sensitivity analysis reported.
assumptions (5)
  • standard math k-means clustering converges and the elbow method selects a valid cluster count
    Used in Section 3.2 and Appendix B to determine the number of clusters and expert groups.
  • domain assumption E5 and Instructor encoder embeddings encode sequence-level knowledge requirements relevant to expert routing
    Used in Section 3.2 to motivate the clustering stage; no direct evidence that embedding similarity aligns with routing quality.
  • domain assumption Cluster centroids computed on training data generalize to evaluation benchmarks
    Section 3.3 assigns test sequences using training-set centroids; Figure 5 only shows cluster distributions on 100 sampled instances per benchmark.
  • domain assumption Adapter-based experts (PESC) preserve the routing and specialization behavior of full-FFN MoE
    Section 3.1 adopts PESC for efficiency; the Limitations section admits full-FFN application is left to future work.
  • ad hoc to paper One-to-one cluster-to-expert-group mapping with |C|=|G| is a sensible inductive bias
    Introduced in Section 3.3 without comparison to alternative mappings such as many-to-one or learned group assignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning." pith.science (2026). https://pith.science/paper/BUPXXQ7J

@misc{pith2026250910513,
  author       = {Pith},
  title        = {Pith review of: Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUPXXQ7J}},
  note         = {Machine review of arXiv:2509.10513}
}
abstract

A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specialization to enhance performance and generalization remains a challenge for MoE, especially in instruction tuning scenarios characterized by significant input heterogeneity. In this work, we propose the Mixture-of-Clustered-Experts (MoCE) to address this limitation through a dual-stage routing mechanism. The first stage in the mechanism performs expert group routing based on sequence-level features, while the second stage activates the top-$k$ experts within the group at the token level. This approach enables the effective partitioning of heterogeneous inputs based on their knowledge requirements, encouraging expert group specialization while maintaining the advantages of token-level routing. We evaluate MoCE across a comprehensive set of benchmarks, demonstrating its consistent superiority over strong baselines and its enhanced generalization capabilities. Detailed analysis further highlights the robustness and effectiveness of MoCE.

Figures

Figures reproduced from arXiv: 2509.10513 by the authors.

Figure 1
Figure 1. The Overall Architecture of MoCE. MoCE consists of two hierarchical stages: (1) Sequence-level expert [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. K-means clustering results based on sequence embeddings from Instructor and E5 models [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance of MoCE applied to Mistral and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance variations across different num [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Cluster prediction results on four evaluation benchmarks, based on 100 sampled instances per benchmark. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. https://doi.org/10.18653/v1/N19-1245 M ath QA : Towards interpretable math word problem solving with operation-based formalisms . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics:...

    doi:10.18653/v1/n19-1245 2019

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  5. [5]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. https://openreview.net/forum?id=IW1PR7vEBf LLM 2vec: Large language models are secretly powerful text encoders . In First Conference on Language Modeling

    2024

  6. [6]

    Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. 2022. A framework for the evaluation of code generation models. https://github.com/bigcode-project/bigcode-evaluation-harness

    2022

  7. [7]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. arXiv preprint arXiv:2407.06204

    arXiv 2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

Show all 53 references
  1. [9]

    Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, and Zhangyang Wang. 2023. https://openreview.net/forum?id=w1hwFUb_81 Sparse moe as the new dropout: Scaling dense and self-slimmable transformers . In The Eleventh International Conference on Learning Representations

    2023

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y

    Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. https://doi.org/10.18653/v1/2024.acl-long.70 D eep S eek M o E :...

    2024 doi

  5. [13]

    Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. 2022. One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.06126

    2022 arXiv

  6. [14]

    Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang. 2023. https://doi.org/10.18653/v1/2023.acl-long.280 Mixture-of-domain-adapters: Decoupling and injecting domain knowledge to pre-trained language models ' memories . In Proceedings of the 61st Annual Meeting of ...

    2023 doi

  7. [15]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. 2024. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Ass...

    2024

  8. [16]

    Dongyang Fan, Bettina Messmer, and Martin Jaggi. 2024. https://openreview.net/forum?id=ebPKyb6r9F TOWARDS AN EMPIRICAL UNDERSTANDING OF MOE DESIGN CHOICES . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    2024

  9. [17]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  10. [18]

    Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. 2023. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312.12379

    2023 arXiv

  11. [19]

    Smith, and Luke Zettlemoyer

    Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.naacl-main.407 DEM ix layers: Disentangling domains for modular language modeling . In Proceedings of the 2022 Conference of the North American Chapter of t...

    2022 doi

  12. [20]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790--2799. PMLR

    2019

  13. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  14. [22]

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. https://openreview.net/forum?id=T5nUQDrM4u Sparse upcycling: Training mixture-of-experts from dense checkpoints . In The E...

    2023

  15. [23]

    Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.304 Beyond distillation: Task-level mixture-of-experts for efficient inference . In Findings of the Association ...

    2021 doi

  16. [24]

    Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, and Chanvichet Vong. 2023. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 5

    2023

  17. [25]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. https://proceedings.mlr.press/v202/longpre23a.html The flan collection: Designing data and methods for effective instruction tuning . I...

    2023

  18. [26]

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1930--1939

    2018

  19. [27]

    Mohammed Muqeeth, Haokun Liu, and Colin Raffel. 2024. https://openreview.net/forum?id=QHzzAU7Qf9 Soft merging of experts with adaptive routing

    2024

  20. [28]

    Oleksiy Ostapenko, Lucas Caccia, Zhan Su, Nicolas Le Roux, Laurent Charlin, and Alessandro Sordoni. 2023. A case study of instruction tuning with mixture of parameter-efficient experts. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following

    2023

  21. [29]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

    2023 arXiv

  22. [30]

    Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. 2024. https://openreview.net/forum?id=jxpsAj7ltE From sparse to soft mixtures of experts . In The Twelfth International Conference on Learning Representations

    2024

  23. [31]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258

    2024 arXiv

  24. [32]

    Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. 2023. Pangu- \ Sigma \ : Towards trillion parameter language model with sparse heterogeneous computing. arXiv preprint arXiv:...

    2023 arXiv

  25. [33]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://doi.org/10.1145/3474381 Winogrande: an adversarial winograd schema challenge at scale . Commun. ACM, 64(9):99–106

    2021 doi

  26. [34]

    Soumajyoti Sarkar, Leonard Lausen, Volkan Cevher, Sheng Zha, Thomas Brox, and George Karypis. 2024. Revisiting smoe language models by evaluating inefficiencies with task specific expert pruning. arXiv preprint arXiv:2409.01483

    2024 arXiv

  27. [35]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. https://openreview.net/forum?id=B1ckMDqlg Outrageously large neural networks: The sparsely-gated mixture-of-experts layer . In International Conference on Learning ...

    2017

  28. [36]

    Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. 2024. https://openre...

    2024

  29. [37]

    Smith, Luke Zettlemoyer, and Tao Yu

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. https://doi.org/10.18653/v1/2023.findings-acl.71 One embedder, any task: Instruction-finetuned text embeddings . In Findings of the Asso...

    2023 doi

  30. [38]

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wen Li, Wen tau Yih, Jason E Weston, and Xian Li. 2024. https://openreview.net/forum?id=nqLAuMOF6n Branch-train-mix: Mixing expert LLM s into a mixture-of-experts LLM...

    2024

  31. [39]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. https://arxiv.org/abs/2210.09261 Challenging big-bench tasks and whether chain-of-thought can solve them . ...

    2022 arXiv

  32. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  33. [41]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022 a . Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533

    2022 arXiv

  34. [42]

    Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022 b . https://doi.org/10.18653/v1/2022.emnlp-main.388 A da M ix: Mixture-of-adaptations for parameter-efficient model tuning . In Proceedings of the 2022 Conf...

    2022 doi

  35. [43]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  36. [44]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and...

    2024 arXiv

  37. [45]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning

    2024

  38. [46]

    Haoyuan Wu, Haisheng Zheng, and Bei Yu. 2024 a . Parameter-efficient sparsity crafting from dense to mixture-of-experts for instruction tuning on general tasks. arXiv preprint arXiv:2401.02731

    2024 arXiv

  39. [47]

    Xun Wu, Shaohan Huang, and Furu Wei. 2024 b . https://openreview.net/forum?id=uWvKBCYh4S Mixture of lo RA experts . In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739

    2024 arXiv

  41. [49]

    Ted Zadouri, Ahmet \"U st \"u n, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. 2024. https://openreview.net/forum?id=EvDeiLv7qc Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning . In The Twelfth International Conf...

    2024

  42. [50]

    Xinyu Zhao, Xuxi Chen, Yu Cheng, and Tianlong Chen. 2023. Sparse moe with language guided routing for multilingual machine translation. In The Twelfth International Conference on Learning Representations

    2023

  43. [51]

    Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang. 2022. Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-experts. Advances in Neural Information Processing Systems, 35:22243--22257

    2022

  44. [52]

    Tong Zhu, Daize Dong, Xiaoye Qu, Jiacheng Ruan, Wenliang Chen, and Yu Cheng. 2024. Dynamic data mixing maximizes instruction tuning for mixture-of-experts. arXiv preprint arXiv:2406.11256

    2024 arXiv

  45. [53]

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.