REVIEW 4 major objections 5 minor 15 references
Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Robustifying only the runner-up expert in a mixture-of-experts model raises adversarial accuracy from 53.7% to 69.1% under PGD on CIFAR-10 at a 1.27-point clean-accuracy cost, and a jointly-trained dual model can exceed the standard…
desk verdict Promising MoE robustness work with a load-bearing ambiguity: the RT-ER loss is defined two different ways, so the headline results cannot be attributed to a single algorithm. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The two objects carrying the argument are the RT-ER loss $L_{\text{rob}} = \max_{\|\delta\|_p \leq \epsilon} \ell_{CE}(F_R(x+\delta), y) + \beta \cdot \ell_{KL}(f_2(x+\delta), f_2(x))$, where $f_2$ is the second-top expert, and the dual model $F_D(x) = (1-\alpha) F_S(x) + \alpha F_R(x)$ with $\alpha \in [1/2, 1]$. The KL term forces an expert the router did not select to keep its output similar on clean and adversarial inputs, which stabilizes training when the router switches experts and hardens the whole model. The certified bound of Theorem 5.5, $\epsilon = \min_{k \neq y} \frac{\alpha(F_R^{(y)}(x) - F_R^{(k)}(x)) + \alpha - 1}{\alpha \sum_i (2 r_{Ri} + a_{Ri}(x)(L_{Ri}^{(y)} + L_{Ri}^{(k)}))}$, shows how the dual model's guaranteed radius grows with the robust MoE's class margin and shrinks with the Lipschitz constants of experts and router; the bound forces $\alpha \geq 1/2$ and justifies directly enlarging the margin through joint training.
What would settle it
Run RT-ER on CIFAR-10 with ResNet18 experts under the same 10-step PGD budget ($\epsilon = 8/255$) and sweep $\beta$ over {1,2,3,4,5,6,7,8,9}; if no $\beta$ both keeps standard accuracy above 77% and robust accuracy above 60%, or if $\beta = 6$ does not reproduce the reported 69.09% robust accuracy, the central claim fails. A second decisive check is to compute the certified radius of Theorem 5.5 pointwise on the trained JTDMoE: if the bound is effectively zero on test points while empirical robust accuracy is around 75%, the margin-Lipschitz mechanism claimed to underlie the method is not what is providing the robustness.
Extended reading notes
Core claim
The central discovery is a component-level asymmetry: in a standard MoE, expert networks are far more attackable than the router, so robustness effort should be spent on experts rather than end-to-end training. The authors show that adding a KL-divergence term to the adversarial-training loss for one additional expert—the one with the second-largest router weight, called the second-top expert—stabilizes the unstable training that plain adversarial training exhibits on MoEs and raises PGD robust accuracy from 53.74% to 69.09% on CIFAR-10, with only a 1.27-point drop in standard accuracy. They then prove a certified robustness radius for any robust MoE (Theorem 5.4) and for the blended dual model (Theorem 5.5), the latter showing that robustness is inherited from the robust MoE and scales with that model's class margin and with the inverse Lipschitz constants of experts and router. Under their bi-level joint training scheme JTDMoE, the dual model simultaneously lifts both standard and robust accuracy beyond the pre-trained combination, and per-class margins improve for every class, in line with the certified bound.
Load-bearing premise
The paper's gains hinge on the KL-penalty coefficient $\beta$ being manually set to 6: values of 1 or 3 give weak robustness, and $\beta = 9$ collapses training to random 10% accuracy, but the paper offers no a priori rule for choosing $\beta$.
Editorial extensions
If this is right
- RT-ER requires robustifying only one additional expert per input, so its training cost stays close to ordinary adversarial training while the robustness gain under PGD is about 15 points on CIFAR-10.
- The dual-model with $\alpha$ in $[0.5, 1]$ provides a continuous, user-selectable trade-off between standard and robust accuracy; at $\alpha = 0.7$ the jointly-trained JTDMoE exceeds the standard MoE's clean accuracy while adding 22.08 points of robust accuracy on CIFAR-10.
- Because the certified bound of Theorem 5.5 shows the dual model's robustness is inherited from the robust MoE, any defense that only hardens the standard MoE cannot improve the certificate.
- The component-level finding that experts are far more vulnerable than routers (RA-E about 3% vs RA-R about 55% on CIFAR-10) implies that future MoE defenses should budget robustness effort per expert rather than applying end-to-end adversarial training.
Reading between the lines
- A testable extension the paper does not explore is applying the second-top-expert KL penalty to other conditional-computation architectures, such as mixture-of-depths layers or routed attention; if the $\beta$-sensitivity transfers, the mechanism is general to routing, not specific to MoE classifiers.
- The certified bound's dependence on the router's Lipschitz constant suggests an untested complementary defense: explicitly regularizing the router's Lipschitz constant during RT-ER could widen the certified radius beyond what expert-only KL regularization achieves.
- Because $\beta = 9$ collapses training, the method as presented requires manual $\beta$ selection for each new dataset or architecture; an adaptive scheduling rule that scales $\beta$ with the KL term's magnitude would make RT-ER practical outside the two settings tested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies adversarial robustness of Mixture-of-Experts (MoE) models. It first presents a component-level vulnerability analysis on CIFAR-10 and TinyImageNet, concluding that expert networks are substantially more attack-susceptible than the router. Based on this, it proposes RT-ER, a robust training loss that augments standard adversarial training with a KL-divergence term intended to make an additional non-selected expert robust. It then introduces a dual-model construction that linearly interpolates a standard MoE and a robust MoE with a smoothing parameter α, and a joint training scheme called JTDMoE that alternates between robustifying the robust MoE and training the dual model. The paper derives certified robustness bounds for the single robust MoE (Theorem 5.4) and for the dual model (Theorem 5.5), and reports experiments on CIFAR-10 and TinyImageNet with ResNet18 and ViT-small experts, including comparisons with standard training, adversarial training (AT), TRADES, and AdvMoE, and an ImageNet/ViT scalability study. The headline empirical claims are that RT-ER raises PGD robust accuracy on CIFAR-10 from 53.74% (AT) to 69.09% with only a 1.27-point standard-accuracy drop, and that JTDMoE at α=0.7 reaches 92.29% standard accuracy and 74.62% robust accuracy, exceeding the standard MoE's clean accuracy while adding about 22 points of robustness.
Significance. If the claims hold, the work is significant: it identifies a structurally important vulnerability in MoE (experts over router), proposes a targeted defense that is computationally cheaper than full adversarial training, and provides a principled interpolation mechanism for trading off robustness and accuracy. The paper also includes several strengths: the code is publicly released, the evaluation spans multiple architectures and datasets, and the component-wise attack evaluation (RA-E vs RA-R) is a useful diagnostic. The theoretical bounds, if correct, would provide a formal justification for the proposed training objectives. However, the significance is currently undermined by a major ambiguity in the definition of the central loss function, by the absence of statistical confidence measures in all reported numbers, and by a proof gap in the dual-model bound. These issues prevent the reader from attributing the headline results to the claimed targeted mechanism.
major comments (4)
- [Section 4.2, Equation (2); Appendix A.3, Equation before Table 9] The RT-ER loss is specified twice with materially different forms. Equation (2) defines Lrob as max CE plus β·ℓKL(f2(x+δ), f2(x)), where f2 is the second-top expert only, and the text and Figure 1 repeatedly emphasize that only one additional expert is robustified. In contrast, Appendix A.3 recalls 'the loss function Lrob used in the RT-ER method' as max CE plus β·Σ_{i=1}^E ℓKL(fRi(x+δ), fRi(x)), i.e., a sum over all experts, and the β ablation in Table 9, including the explanation for the β=9 collapse to 10% accuracy, is explicitly based on this summed version. These two objectives have different computational costs and different regularization semantics. The paper never states which objective was used to produce Tables 2 and 3 and the headline CIFAR-10 comparison. If the experiments used the summed loss, the claims that RT-ER 'requires training only one additional expert' and introduces minimal training overhead are unsupported, and the observed gains may be due to regularizing every expert rather than the proposed targeted mechanism. If the experiments used Equation (2), then the Appendix A.3 loss definition and the β study are not about the actual method. Either way, the central empirical claim cannot be evaluated from the text alone. The authors must reconcile these definitions, state exactly which loss was minimized for each reported result, and re-run or re-report experiments as needed.
- [Appendix A.3, Table 9] The KL-penalty coefficient β is selected after observing test performance, and the reported search suggests that the method is highly sensitive to it. Among β ∈ {1,3,6,9}, β=6 is the only value that neither yields relatively weak robust accuracy (β=1,3) nor total training collapse (β=9 gives 10% accuracy on all metrics, i.e., random guessing). No a priori or validation-based selection rule is given, and no sensitivity analysis is provided for other datasets, attack budgets, or architectures. Since β is a free parameter tuned on the test set, the magnitude of the reported improvements may be optimistically biased. Please provide a principled selection procedure (e.g., validation-set selection), report results for multiple seeds, or demonstrate that the ranking of methods is stable across a range of β.
- [Tables 1-10, Section 6] All reported accuracies come from single training runs with no standard deviations, confidence intervals, or number of seeds. Several of the paper's key comparisons are small: for example, in Table 4 the TinyImageNet AutoAttack robust accuracy improves from 51.27% to 52.91% (1.64 points), and Table 5 reports per-class margin improvements as low as 1.25%. Given the training instability documented in Figure 2 and the collapse observed at β=9, these differences may be within run-to-run variation. The authors should report mean and standard deviation over at least three independent seeds for the main tables, or otherwise justify why single runs are sufficient.
- [Appendix A.1, proof of Theorem 5.5] The proof of Theorem 5.5 does not establish the displayed bound. It invokes Theorem 3.5 of Bai et al. (2024a) and then states that 'from the previous proof, we already derived the Lipschitz constant for the robust MoE,' but the denominator in Equation (9) contains per-class Lipschitz constants L_Ri^(y) and L_Ri^(k), which are not defined in Assumption 5.3 (only a single LRi for the y-th output is defined). Moreover, the transition from the Lipschitz bound of a scalar output to the Lipschitz constant of the margin F_R^(y) − F_R^(k) is not shown; the margin's Lipschitz constant is not simply the sum of per-class Lipschitz constants unless additional structure is assumed. A full derivation starting from Assumption 5.3 is needed, or the theorem must be restated with the missing assumptions made explicit.
minor comments (5)
- [Section 6.1, baseline list] There is a typo in the baseline list: 'expters robustification' should be 'experts robustification.'
- [Section 6.2, text after Table 3] The sentence 'RT-ER improves RA-E by over 70% and 17% on ResNet18 and ViT-small, respectively' should read 'by over 70 and 17 percentage points,' because the underlying numbers are differences in percentages, not relative improvements.
- [Appendix A.6, Table 10] The ImageNet experiment lacks setup details: no specification of the dataset subset, number of experts, attack budget, training epochs, or whether the ViT was pre-trained. This makes the scalability claim difficult to reproduce; please add the full experimental configuration.
- [Appendix A.4, Figure 6] The x-axis labels in Figure 6 appear corrupted ('0.5 1' repeated); the intended α values from 0.5 to 1.0 should be displayed cleanly.
- [Section 5.2, Equation (10)] The bi-level formulation 'min_{ΘS,ΘR} ℓCE(FD(x), y) subject to min_{ΘR} Lrob' is not a standard constrained optimization problem as written; the constraint 'min_{ΘR} Lrob' is an optimization problem, not a constraint set. Please rephrase using standard bilevel or alternating-minimization notation.
Circularity Check
No load-bearing circularity; the only self-referential element is the margin-based 'verification' of Theorem 5.5, which is a consistency check rather than an independent test.
-
other
[Section 6.4, 'Margin Comparison' paragraph after Table 5; relates to Theorem 5.5 and Eq. (9)]
"These results empirically verify Theorem 5.5, demonstrating that margin enhancement contributes to JTDMoE’s superior robust accuracy (RA)."
The certified bound in Eq. (9) contains the robust MoE margin F_R^y(x) - F_R^k(x) in its numerator by construction, with only Lipschitz/router terms in the denominator. The empirical 'verification' measures exactly this margin (Table 5) and reports that JTDMoE improves it; the Lipschitz constants in the denominator are not measured. Thus the observation restates a variable already embedded in the theorem's bound rather than independently testing the theorem. This is a mild self-referential validation framing, though the theorem's derivation itself rests on Assumption 5.3 and the external result of Bai et al. (2024a), so the derivation chain is not circular.
full rationale
The paper's core derivation chain is not circular. Theorem 5.4 is a direct Lipschitz-margin bound proved from Assumption 5.3 in Appendix A.1. Theorem 5.5 is explicitly motivated by and proved using Theorem 3.5 of Bai et al. (2024a), an external result, and the proof in Appendix A.1 builds the dual-model bound from the single-MoE Lipschitz constants. RT-ER's loss in Eq. (2) is a proposed training objective, and beta is a hyperparameter selected by ablation (Table 9), not a parameter fitted to the headline RA numbers. Self-citations (Wang et al. 2020a; Li et al. 2023) are background and not load-bearing. The only self-referential element is the margin comparison in Section 6.4: because Eq. (9)'s bound is a function of the margin, observing margin gains is a consistency check, not an independent 'verification' of Theorem 5.5. Separately, a non-circular reproducibility concern exists: Eq. (2) defines the KL term only for the second-top expert f2, while Appendix A.3 recalls Lrob with a sum over all experts and bases the beta=9 collapse explanation on that summed form; the paper never states which objective produced Tables 2-3. That is an attribution/correctness issue, not a circular reduction, so it does not raise the circularity score.
Assumptions & free parameters
free parameters (2)
- beta (KL regularization weight) =
6 (grid: 1, 3, 6, 9; beta=9 collapses to 10% accuracy)
- alpha (dual-model smoothing weight) =
0.7 for main JTDMoE results; swept 0.5 to 1.0
assumptions (3)
- domain assumption Each expert and the router are ell_p-Lipschitz continuous (Assumption 5.3).
- domain assumption The expert-targeted attack with fixed router weights measures the true vulnerability of experts; 98% of such perturbations keep the same routing.
- ad hoc to paper Robustifying the second-top expert across minibatches is sufficient to make all experts robust.
Cite this review
Pith. "Pith review of Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach." pith.science (2026). https://pith.science/paper/TO4INOO4
@misc{pith2026250206832,
author = {Pith},
title = {Pith review of: Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/TO4INOO4}},
note = {Machine review of arXiv:2502.06832}
}
read the original abstract
Mixture of Experts (MoE) have shown remarkable success in leveraging specialized expert networks for complex machine learning tasks. However, their susceptibility to adversarial attacks presents a critical challenge for deployment in robust applications. This paper addresses the critical question of how to incorporate robustness into MoEs while maintaining high natural accuracy. We begin by analyzing the vulnerability of MoE components, finding that expert networks are notably more susceptible to adversarial attacks than the router. Based on this insight, we propose a targeted robust training technique that integrates a novel loss function to enhance the adversarial robustness of MoE, requiring only the robustification of one additional expert without compromising training or inference efficiency. Building on this, we introduce a dual-model strategy that linearly combines a standard MoE model with our robustified MoE model using a smoothing parameter. This approach allows for flexible control over the robustness-accuracy trade-off. We further provide theoretical foundations by deriving certified robustness bounds for both the single MoE and the dual-model. To push the boundaries of robustness and accuracy, we propose a novel joint training strategy JTDMoE for the dual-model. This joint training enhances both robustness and accuracy beyond what is achievable with separate models. Experimental results on CIFAR-10 and TinyImageNet datasets using ResNet18 and Vision Transformer (ViT) architectures demonstrate the effectiveness of our proposed methods. The code is publicly available at https://github.com/TIML-Group/Robust-MoE-Dual-Model.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Alam, M., Datta, S., Mukhopadhyay, D., Mondal, A., and Chakrabarti, P. P. Resisting adversarial attacks in deep neural networks using diverse decision boundaries. arXiv preprint arXiv:2208.08697,
-
[7]
Performance evaluation of JTDMoE under different val- ues of α on the CIFAR-10 test dataset. We use the dual-model performance at α = 0.7 as the baseline, depicted by green squares in this figure. Standard accuracy (SA) and robust accuracy (RA) under a 50-step PGD attack are reported, with models trained us- ing a 10-step PGD attack. The results demonstra...
work page 2009
-
[8]
Car- bon emissions and large neural network training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, J. Car- bon emissions and large neural network training. arXiv preprint arXiv:2104.10350,
-
[9]
From sparse to soft mixtures of experts
Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951,
-
[11]
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,
-
[13]
Mixture of experts in image classification: What’s the sweet spot? arXiv preprint arXiv:2411.18322,
Videau, M., Leite, A., Schoenauer, M., and Teytaud, O. Mixture of experts in image classification: What’s the sweet spot? arXiv preprint arXiv:2411.18322,
-
[14]
11 Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach A. Supplementary Material In this supplementary material, we provide the proofs of Theorems 5.4 and 5.5 in Section A.1. Additional experimen- tal results are organized as follows: single MoE experiments are reported in Section A.3, dual-model experiments based on pre-traine...
work page 2021
-
[2016]
Mixture- of-experts for semantic segmentation of remoting sensing image
He, S., Cheng, Q., Huai, Y ., Zhu, Z., and Ding, J. Mixture- of-experts for semantic segmentation of remoting sensing image. In International Conference on Image Processing and Artificial Intelligence (ICIPAl 2024), volume 13213, pp. 478–483. SPIE,
work page 2024
Show all 15 references
-
[2017]
Smith, L. N. Cyclical learning rates for training neural net- works. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464–472. IEEE,
2017
-
[2018]
B., and Swami, A
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pp. 372–387. IEEE,
2016
-
[2019]
J., Shlens, J., and Szegedy, C
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv:1412.6572,
-
[2020]
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,
2009
-
[2021]
Towards deep learning models resistant to adver- sarial attacks
Madry, A. Towards deep learning models resistant to adver- sarial attacks. arXiv preprint arXiv:1706.06083,
-
[2022]
G., Kim, A., and Sojoudi, S
Bai, Y ., Anderson, B. G., Kim, A., and Sojoudi, S. Improv- ing the accuracy-robustness trade-off of classifiers via adaptive smoothing. SIAM Journal on Mathematics of Data Science, 6(3):788–814, 2024a. Bai, Y ., Zhou, M., Patel, V . M., and Sojoudi, S. Mixednuts: Training-fre...
2017 arXiv
-
[2023]
A., Stimberg, F., Wiles, O., and Mann, T
Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, O., and Mann, T. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946,
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.