Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

Haixi Fan; J. Marius Z\"ollner; Konstantin Ditschuneit; Svetlana Pavlitska

REVIEW 4 major objections 4 minor 34 references

Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Inserting a single sparse mixture-of-experts layer in a deep CNN stage improves adversarial robustness under PGD and AutoPGD attacks, but only when the network is adversarially trained.

desk verdict A useful empirical study of block/conv-level MoE in adversarially trained ResNets, with a plausible robustness gain but missing attack epsilon and seed variance; worth engaging after those are fixed. read the letter →

arxiv 2509.05086 v1 pith:4U5KDJHH submitted 2025-09-05 cs.CV cs.LG

Svetlana Pavlitska , Haixi Fan , Konstantin Ditschuneit , J. Marius Zöllner This is my paper

classification cs.CVcs.LG

keywords adversarialrobustnesssparsemixture-of-expertsResNetCIFAR-100trainingroutingcollapseexpertspecializationAutoPGD

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse mixture-of-experts (MoE) layers activate only a small subset of expert sub-networks per input, adding capacity without raising inference cost. The paper tests whether inserting such layers into ResNets trained on CIFAR-100 can harden them against adversarial examples. Its main finding is that a single MoE layer placed in the deepest stage consistently raises robust accuracy under PGD-20 and AutoPGD attacks when the model is adversarially trained, while leaving clean accuracy roughly unchanged. A second finding is that the switch balancing loss makes routing collapse onto a few experts, and those overused experts can be more robust on their own than the full gated model, suggesting that concentrated adversarial training creates robust subpaths. The paper frames sparse MoEs as an architectural lever for robustness rather than a method that works under ordinary training.

What carries the argument

The load-bearing object is the sparse MoE layer inserted into a CNN: either BlockMoE, which replaces an entire residual block so each expert is itself a BasicBlock or BottleneckBlock, or ConvMoE, which replaces a single convolutional layer. A gate network (GAP-FC or Conv-GAP) scores experts for each input and only the top-k contribute, so capacity grows with the number of experts while per-input FLOPs stay nearly flat. The argument runs through the interaction of this routing mechanism with PGD-7 adversarial training and with the two balancing losses: entropy loss spreads inputs across experts and gives the best full-model robustness, while switch loss collapses routing and produces individu

What would settle it

Rerun the ResNet-18 and ResNet-50 baselines and MoE configurations with 3–5 seeds, using a stated l-infinity epsilon (for example 8/255), and report mean and spread of PGD-20 and AutoPGD accuracy; if the MoE advantage over the dense baseline is not consistently positive across seeds, the central claim collapses.

Watch

Extended reading notes

Core claim

The paper claims that replacing a single residual block or convolutional layer in a deep ResNet stage with a sparse MoE layer—four experts with top-k routing—yields consistent robustness gains under PGD-20 and AutoPGD attacks when combined with PGD-7 adversarial training on CIFAR-100. Block-level MoEs (replacing the second BasicBlock of conv5_x) give the most consistent improvements, and the best full-model configuration uses top-2 routing, the GAP-FC gate, and an entropy-based balancing loss. The paper also reports that the switch loss, while intended to balance experts, instead drives routing to collapse onto a small subset; under adversarial training these heavily-used experts become indi

Load-bearing premise

The paper reports each configuration from a single training run and never states the attack strength used for PGD-20, AutoPGD, or adversarial training, so the claimed MoE improvement assumes the measured gaps are not seed noise and that all models faced equally strong attacks.

Editorial extensions

If this is right

Adversarially trained ResNets on CIFAR-100 can gain robust accuracy under PGD and AutoPGD simply by placing one MoE layer in the deepest stage, at no extra inference cost since only top-k experts run.
BlockMoE (replacing a residual block) is more reliable than ConvMoE (replacing a conv layer), and the best position is the second BasicBlock of conv5_x.
Entropy-based balancing gives the best robustness-accuracy tradeoff for the full model; switch loss collapses routing but creates individual experts that outperform the full MoE under attack.
The robustness benefit of MoE layers is confined to adversarially trained models; under normal training the gains are negligible.
Increasing the number of experts tends to improve both clean and adversarial accuracy up to a moderate point (typically 8–16 experts), beyond which performance levels off or decreases; top-2 routing offers the best tradeoff.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If robust subpaths really form inside a collapsed router, a gate that selects the most robust expert per input at inference time—or after adversarial fine-tuning of expert choice—could beat the full MoE; the paper stops at reporting the effect and lists robustness-aware routing as future work.
The switch-loss result behaves like an implicit lottery-ticket search: adversarial training on a collapsed route finds a robust subnetwork. A direct test would be to prune all but the robust experts and measure whether the small subnetwork retains the robustness of the full MoE.
Because the paper never states the l-infinity epsilon for its attacks, the size of the MoE advantage is not portable across settings. A useful extension is to measure the MoE gain across a sweep of perturbation budgets and across multiple seeds and architectures to learn where the gain comes from and how stable it is.
The robustness gain appearing only under adversarial training suggests the MoE layers do not add robustness by themselves; they change how the model fits adversarial examples. If correct, similar gains might appear in other conditional-computation architectures such as early-exit or dynamic-depth networks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper does something genuinely new: it inserts sparse MoE layers at the block and conv level in ResNet-18/50, trains with PGD-7, and reports a few-point gain in PGD-20/AutoPGD robust accuracy on CIFAR-100 without losing clean accuracy. The best configurations (top-2, entropy loss, GAP-FC gate, deep placement) are consistent enough across the tradeoff figures that I believe the effect is real, even if the paper doesn't say so with error bars.

What it does well: the ablations are thorough. It compares two MoE granularities, three auxiliary losses, two gate designs, several positions, and expert counts from 1 to 32. The routing-collapse analysis (Figure 7) is clearly presented, and the observation that switch-loss experts become individually robust while the full gate does not is a nice, falsifiable finding. The code is public, and the entropy/KL equivalence is a simple identity correctly stated.

The soft spots are real and in proportion. Most importantly, the paper never states the l-infinity epsilon used for the attacks or for PGD-7 adversarial training. That is a basic missing detail in an adversarial robustness paper; without it the numbers are not reproducible. Second, there are no multi-seed runs, so the claimed 'consistent improvements' could be partly seed noise. Some AutoPGD gaps are on the order of 0.5pp, which is within noise for single runs. I'd want at least 2-3 seeds and standard deviations for the headline configurations.

The stress-test worry about gradient masking from top-k non-smoothness is worth taking seriously, but I think it is not a fatal flaw. The router uses softmax probabilities masked to top-k, so gradients flow through the softmax even when the selected set is fixed. AutoPGD is also much less susceptible to masking than plain PGD. Still, a black-box or Square attack would strengthen the claim, and the authors' own Section 4.3.2 shows the learned gate is not the robust part. That is more consistent with routing collapse concentrating training on a few experts than with pure masking, but the missing epsilon makes it hard to fully rule out.

The citation pattern is fine: the self-citation to the GAP-FC gate is an architecture component, not the claimed result, and the related work covers the relevant transformer and CNN MoE robustness literature.

Who should read it: people working on adversarial training or MoE for vision. It is not a breakthrough, but it is a solid empirical study with a testable claim about robust subpaths. I would send it to peer review, conditionally, with a request for epsilon disclosure, multi-seed statistics, and preferably one non-gradient attack.

Read it carefully, then decide.

Referee Report

4 major / 4 minor

Summary. The paper studies the effect of inserting sparse mixture-of-experts (MoE) layers into CNNs on adversarial robustness. Two MoE layer types are proposed: BlockMoE (replaces a residual block) and ConvMoE (replaces a single convolutional layer). Experiments on CIFAR-100 with ResNet-18 and ResNet-50 under normal training and PGD-7 adversarial training report robustness under PGD-20 and AutoPGD attacks. The authors claim that inserting a single MoE layer in deeper stages leads to consistent robustness improvements under adversarial training, and they observe that switch-loss training causes routing collapse, with some individual fixed experts becoming more robust than the full gated model. Code is provided.

Significance. If the claimed effect is genuine, the work offers a simple architectural modification that improves adversarial robustness without adding inference cost, and the discovery of robust subpaths inside a collapsed router is an interesting phenomenon. The paper ships code and makes a falsifiable empirical prediction. However, the current evidence is not sufficient to establish the central claim: the attack perturbation bound is never stated, all comparisons are single-run, and the non-smooth top-k router raises a concrete gradient-masking concern that is not addressed. These are load-bearing issues for the headline conclusion.

major comments (4)

[Section 4.1] The perturbation bound epsilon for PGD-20, AutoPGD, and PGD-7 adversarial training is never stated. The text only says 'l-infinity limited'. Without a numerical epsilon (e.g., 8/255), the robustness percentages in Figure 4 are not interpretable, and one cannot verify that all models were attacked at the same strength. Please specify epsilon for every attack and for adversarial training, and confirm it is identical across baselines and MoE variants.
[Section 4.2 / Figure 4] All reported comparisons are single runs; no seeds, error bars, or statistical tests are provided. Many differences are extremely small on the absolute scale of the relevant subplots (e.g., normal-training PGD accuracy in Figure 4 ranges over 0–0.5% and AutoPGD over 2–4%, with baseline and MoE points differing by much less than 1%). The abstract's 'consistent improvements' claim cannot be distinguished from seed noise. Provide at least 3–5 independent training runs per configuration, report mean±std, and state whether the improvement holds for each seed.
[Section 3.1 and Section 4.2] The top-k gate is non-smooth: in Eq. (1), g_i(x)=0 for unselected experts, making the forward pass piecewise constant in the router logits. PGD-20 and AutoPGD are white-box gradient attacks; they may stop prematurely when the router assignment would need to change to find an adversarial example. This is a classic gradient-masking failure mode, and the paper provides no black-box or randomized attack to rule it out. The finding in Section 4.3.2 that fixed experts outperform the full MoE is consistent with the router being a hard-to-optimize weak point. Please include an independent evaluation (e.g., transfer attacks from a dense surrogate, or a query-based attack such as Square Attack) or otherwise demonstrate that the observed gains are not an artifact of non-differentiable routing.
[Section 4.3.2] The claim that fixed experts 'outperform the baseline by up to 7.5pp' relies on the same single-run setup and does not specify the attack and epsilon used for that comparison, nor how the 'robust experts' are selected (the brown dots in Figure 8). Since this is a central secondary finding, please clarify the selection criterion, report the accuracy of all experts (or a summary), and provide variance over seeds.

minor comments (4)

[Section 4.1] The text says 'We report the average precision on all classes', but the experiments report classification accuracy. Please clarify whether this is mean per-class accuracy or average precision, and adjust terminology.
[Section 3.4] The description of the switch loss as 'cross-entropy between the empirical distribution of expert assignments and a uniform target distribution' is imprecise; the Switch Transformer auxiliary loss is N * sum_i f_i * P_i, which is not a cross-entropy to a uniform distribution. This matters for interpreting the routing-collapse results in Section 4.3.1.
[Figure 4] The y-axis limits vary dramatically across subplots (e.g., 0–0.5 vs 16–24 for PGD accuracy). This makes visual comparisons across training regimes misleading. Please use consistent scales or annotate the axis ranges more clearly.
[Section 4.2] The text says 'entropy loss consistently outperforms the switch loss in terms of adversarial robustness', but later states that under PGD-7 training both losses improve robustness as the number of experts increases. Please reconcile these statements or clarify which configurations the 'consistently' claim covers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: the paper's claims are direct empirical measurements; self-citations supply only architectural components.

full rationale

The paper does not contain a derivation chain that reduces to its own inputs. The central claims—that inserting a single MoE layer improves adversarial robustness, and that routing collapse under switch loss can create robust fixed experts—are presented and supported as direct measurements on trained models, not as predicted consequences of a fitted parameter. The KL/entropy equivalence in Section 3.4 is a mathematical identity (L_KL = -H(P) + const), not a circular construction. The only self-citations are to the authors' prior work [22] for the GAP-FC gate, the KL loss, and the term 'dying expert phenomenon'; these supply architectural components and vocabulary, not the target robustness result. The paper does not invoke a self-authored uniqueness theorem, does not smuggle in an ansatz via citation, and does not rename a known result as a new contribution. The conclusion's own limitation statement—that gains are confined to adversarially trained settings and that gating may not exploit the most robust paths—is an honest acknowledgment of empirical scope, not a circularity. Concerns about unstated l-infinity bounds or single-seed runs are experimental-rigor risks, not circularity. Therefore no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper is empirical and does not fit constants to data for a derivation; free parameters are hyperparameters swept in the study. The central claim rests on the assumption that the single-seed runs and unspecified attack epsilon yield fair comparisons.

free parameters (2)

Number of experts N = 4 (main comparisons)
Chosen for headline experiments; swept up to 32 in Section 4.2 ablation.
Number of activated experts k = 2 (best setup)
Selected as best after sweeping k=1..4; affects the sparsity and the robustness results.

assumptions (4)

standard math KL divergence equals entropy difference plus a constant when the target distribution Q is uniform (Section 3.4)
Derived in the paper; used to justify the entropy loss as a substitute for KL loss.
domain assumption PGD-7 adversarial training is a sufficient proxy for robustness, and the attack epsilon is the same across all models
The paper never states the perturbation bound epsilon; all comparisons assume identical attack settings.
domain assumption CIFAR-100 performance transfers to other vision benchmarks
Only CIFAR-100 is used; the conclusion of 'consistent improvements' is generalized beyond this single dataset.
domain assumption The GAP-FC gate from [22] is a suitable gating design for robustness comparisons
The main results use GAP-FC gating introduced in the authors' previous work; alternative gates are only briefly ablated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers." pith.science (2026). https://pith.science/paper/4U5KDJHH

@misc{pith2026250905086,
  author       = {Pith},
  title        = {Pith review of: Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4U5KDJHH}},
  note         = {Machine review of arXiv:2509.05086}
}

read the original abstract

Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.

Figures

Figures reproduced from arXiv: 2509.05086 by the authors.

**Figure 2.** Gate architectures: GAP-FC averages over each chan [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** ). Qualitatively, the same behavior is shown when half of all experts are active. However, if the number of activated experts k is decoupled from the total number of experts in the MoE layers, the parameter counts decouple from the FLOPs. This scaling behavior motivates the usage of sparse MoE layers in this work. Comparison to the baseline: Across both ResNet-18 and ResNet-50 architectures, adversarial training con… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Robustness-accuracy trade-off for the normally and adversarially trained baselines and models with MoE layers 2 BlockMoE [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 6.** Figure 6: Robustness-accuracy trade-off for different positions and [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 5.** Figure 5: Robustness-accuracy trade-off for different positions and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 7.** Figure 7: Distribution of test inputs to individual experts during training for a [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Accuracy of the baseline (green), a model with an MoE layer (blue line), and models with a single fixed expert (dots) for normal [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 26 canonical work pages

[1]

Moe-rbench: Towards building reliable language models with sparse mixture-of-experts

Guanjie Chen, Xinyu Zhao, Tianlong Chen, and Yu Cheng. Moe-rbench: Towards building reliable language models with sparse mixture-of-experts. InInternational Conference on Machine Learning (ICML), 2024. 2

work page 2024
[2]

Patch-level routing in mixture-of-experts is provably sample-efficient for con- volutional neural networks

Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. Patch-level routing in mixture-of-experts is provably sample-efficient for con- volutional neural networks. InInternational Conference on Machine Learning (ICML), 2023. 2

work page 2023
[3]

Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks

Francesco Croce and Matthias Hein. Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks. InInternational Conference on Ma- chine Learning (ICML), 2020. 4

work page 2020
[4]

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024. 2

arXiv 2024
[5]

Truong Do, Le Khiem, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Binh Nguyen, Chenghao Liu, Savitha Ra- masamy, Xiaoli Li, and Steven C. H. Hoi. Hyperrouter: To- wards efficient training and inference of sparse mixture of experts. InProceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6...

work page 2023
[6]

Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fe- dus, Maarten P

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fe- dus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kath- leen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V . Le, Yonghui Wu, and...

work page 2022
[7]

Learning Factored Representations in a Deep Mixture of Ex- perts

David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. Learning Factored Representations in a Deep Mixture of Ex- perts. 2013. 2

work page 2013
[8]

Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.The Journal of Machine Learning Research, 2022

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.The Journal of Machine Learning Research, 2022. 2, 3, 5

work page 2022

Show all 34 references

[9]

The lottery ticket hy- pothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

Jonathan Frankle and Michael Carbin. The lottery ticket hy- pothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018. 8

2018 arXiv
[10]

Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks

Yonggan Fu, Qixuan Yu, Yang Zhang, Shang Wu, Xu Ouyang, David Cox, and Yingyan Lin. Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks. InConference on Neural Information Processing Systems (NIPS/NeurIPS),
[11]

Explaining and Harnessing Adversarial Examples

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. InInter- national Conference on Learning Representations (ICLR),
[12]

Sparse dnns with improved adversarial robustness

Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dnns with improved adversarial robustness. InConference on Neural Information Processing Systems (NIPS/NeurIPS), pages 240–249, 2018. 2

2018
[13]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016. 4

2016
[14]

Jacobs, Michael I

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts.Neu- ral computation, 3(1), 1991. 2

1991
[15]

Robustifying routers against in- put perturbations for sparse mixture-of-experts vision trans- formers.IEEE Open Journal of Signal Processing, 2025

Masahiro Kada, Ryota Yoshihashi, Satoshi Ikehata, Rei Kawakami, and Ikuro Sato. Robustifying routers against in- put perturbations for sparse mixture-of-experts vision trans- formers.IEEE Open Journal of Signal Processing, 2025. 2

2025
[16]

Learning multiple layers of features from tiny images

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 4

2009
[17]

Gshard: Scaling giant models with conditional com- putation and automatic sharding

HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional com- putation and automatic sharding. InInternational Confer- ence on Learning Representations (ICLR), 2021. 1

2021
[18]

Modeling task relationships in multi-task learning with multi-gate mixture-of-experts

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. InInternational Conference on Knowledge Discovery & Data Mining, 2018. 2

2018
[19]

Towards Deep Learn- ing Models Resistant to Adversarial Attacks.International Conference on Learning Representations (ICLR), 2018

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learn- ing Models Resistant to Adversarial Attacks.International Conference on Learning Representations (ICLR), 2018. 1, 4

2018
[20]

Choosing smartly: Adaptive multimodal fusion for object detection in changing environments

Oier Mees, Andreas Eitel, and Wolfram Burgard. Choosing smartly: Adaptive multimodal fusion for object detection in changing environments. InIEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS). IEEE, 2016. 2

2016
[21]

Is temper- ature sample efficient for softmax gaussian mixture of ex- perts? InInternational Conference on Machine Learning (ICML), 2024

Huy Nguyen, Pedram Akbarian, and Nhat Ho. Is temper- ature sample efficient for softmax gaussian mixture of ex- perts? InInternational Conference on Machine Learning (ICML), 2024. 2

2024
[22]

Marius Z¨ollner

Svetlana Pavlitska, Christian Hubschneider, Lukas Struppek, and J. Marius Z¨ollner. Sparsely-gated mixture-of-expert lay- ers for cnn interpretability. InInternational Joint Conference on Neural Networks (IJCNN), 2023. 1, 2, 3, 6

2023
[23]

Mar- ius Z ¨ollner

Svetlana Pavlitskaya, Christian Hubschneider, Michael We- ber, Ruby Moritz, Fabian H¨uger, Peter Schlicht, and J. Mar- ius Z ¨ollner. Using mixture of expert models to gain in- sights into semantic segmentation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition ...

2020
[24]

On the adversarial robustness of mixture of experts.CoRR, abs/2210.10253,

Joan Puigcerver, Rodolphe Jenatton, Carlos Riquelme, Pran- jal Awasthi, and Srinadh Bhojanapalli. On the adversarial robustness of mixture of experts.CoRR, abs/2210.10253,

arXiv
[25]

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next- Generation AI Scale

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next- Generation AI Scale. InInternational Conference on Ma- chine Lear...

2022
[26]

Scaling vision with sparse mix- ture of experts

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr ´e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mix- ture of experts. InConference on Neural Information Pro- cessing Systems (NIPS/NeurIPS), 2021. 2, 3, 5

2021
[27]

Le, Geoffrey E

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations (ICLR), 2017. 1, 2, 3, 6

2017
[28]

Goodfellow, and Rob Fergus

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. InInternational Conference on Learning Representations (ICLR), 2014. 1

2014
[29]

Convo- luted mixture of deep experts for robust semantic segmenta- tion

Abhinav Valada, Ankit Dhall, and Wolfram Burgard. Convo- luted mixture of deep experts for robust semantic segmenta- tion. InInternational Conference on Intelligent Robots and Systems (IROS) - Workshops, 2016. 2

2016
[30]

Gonzalez

Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E. Gonzalez. Deep mixture of experts via shallow embedding. InCon- ference on Uncertainty in Artificial Intelligence, UAI. AUAI Press, 2019. 1, 2, 3

2019
[31]

Remoe: Fully differentiable mixture-of-experts with relu routing.CoRR, abs/2412.14711, 2024

Ziteng Wang, Jianfei Chen, and Jun Zhu. Remoe: Fully differentiable mixture-of-experts with relu routing.CoRR, abs/2412.14711, 2024. 2

2024 arXiv
[32]

Limoe: Mixture of lidar representation learners from automotive scenes.CoRR, abs/2501.04004,

Xiang Xu, Lingdong Kong, Hui Shuai, Liang Pan, Ziwei Liu, and Qingshan Liu. Limoe: Mixture of lidar representation learners from automotive scenes.CoRR, abs/2501.04004,

arXiv
[33]

Learning a mixture of granularity-specific experts for fine- grained categorization

Lianbo Zhang, Shaoli Huang, Wei Liu, and Dacheng Tao. Learning a mixture of granularity-specific experts for fine- grained categorization. InIEEE International Conference on Computer Vision (ICCV), 2019. 2

2019
[34]

Robust mixture-of-expert training for convo- lutional neural networks

Yihua Zhang, Ruisi Cai, Tianlong Chen, Guanhua Zhang, Huan Zhang, Pin-Yu Chen, Shiyu Chang, Zhangyang Wang, and Sijia Liu. Robust mixture-of-expert training for convo- lutional neural networks. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023. 1, 2, 3

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Moe-rbench: Towards building reliable language models with sparse mixture-of-experts

Guanjie Chen, Xinyu Zhao, Tianlong Chen, and Yu Cheng. Moe-rbench: Towards building reliable language models with sparse mixture-of-experts. InInternational Conference on Machine Learning (ICML), 2024. 2

work page 2024

[2] [2]

Patch-level routing in mixture-of-experts is provably sample-efficient for con- volutional neural networks

Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. Patch-level routing in mixture-of-experts is provably sample-efficient for con- volutional neural networks. InInternational Conference on Machine Learning (ICML), 2023. 2

work page 2023

[3] [3]

Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks

Francesco Croce and Matthias Hein. Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks. InInternational Conference on Ma- chine Learning (ICML), 2020. 4

work page 2020

[4] [4]

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024. 2

arXiv 2024

[5] [5]

Truong Do, Le Khiem, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Binh Nguyen, Chenghao Liu, Savitha Ra- masamy, Xiaoli Li, and Steven C. H. Hoi. Hyperrouter: To- wards efficient training and inference of sparse mixture of experts. InProceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6...

work page 2023

[6] [6]

Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fe- dus, Maarten P

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fe- dus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kath- leen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V . Le, Yonghui Wu, and...

work page 2022

[7] [7]

Learning Factored Representations in a Deep Mixture of Ex- perts

David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. Learning Factored Representations in a Deep Mixture of Ex- perts. 2013. 2

work page 2013

[8] [8]

Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.The Journal of Machine Learning Research, 2022

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.The Journal of Machine Learning Research, 2022. 2, 3, 5

work page 2022