Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Improving Routing in Sparse Mixture of Experts with Graph of Tokens

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Routing in sparse mixture-of-experts models becomes more stable when each token's expert score is replaced by a similarity- or attention-weighted average of other tokens' scores; the paper proves this lowers selection entropy and reports…

desk verdict A cheap, effective router-smoothing method for SMoE whose advertised entropy proof only covers a degenerate limit, not the actual algorithm. read the letter →

arxiv 2505.00792 v1 pith:OIWPZEJE submitted 2025-05-01 cs.LG

Tam Nguyen , Ngoc N. Tran , Khai Nguyen , Richard G. Baraniuk This is my paper · ORCID
classification cs.LG
keywords sparsemixtureofexpertsroutingfluctuationtokenexpertselectionentropyprobabilisticgraphicalmodelsimilarity-awareattention-awarerobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sparse mixture-of-experts (SMoE) models assign each token to experts independently, through the router score $r(u_i)$ and nothing else; the paper argues this per-token independence is why routing decisions keep changing at the end of training, when up to a third of tokens still switch experts between final epochs. It proposes two nearly drop-in fixes, Similarity-Aware and Attention-Aware routing, in which the score used for token $u_i$ becomes a weighted mixture $\sum_j S[i,j] r(u_j)$ of its neighbors' router scores, with weights given by token similarity or by the attention matrix. The paper proves that this mixing lowers the entropy of expert selection (in the limits $\tau \to 0$ or $\sigma \to 0$), making indecisive tokens more confident, and reports reduced routing fluctuation, lower perplexity, and higher accuracy across language, vision, and fine-tuning benchmarks at roughly 5% extra compute. If correct, stable routing is available as a small router-level modification to existing SMoE layers rather than a new architecture.

What carries the argument

The load-bearing mechanism is the replacement of the standalone router score with a weighted average over tokens: $p_i = \sum_j S[i,j] r(u_j)$, computed either from token similarities (Definition 2, Eq. 8) or from the attention matrix restricted to the head with the lowest average attention entropy (Definition 5, Eq. 16). This coupling is what breaks the conditional independence of expert selections, and it is also what makes the entropy argument work: writing the mixed selection as a mixture of the neighbors' categorical distributions and applying the decomposition $H(e_i^s, t_i) = H(t_i) + \sum_j s(i,j) H(e_j) \geq H(e_i^s)$ gives the Proposition 1 bound, with $H(t_i)$ vanishing as the weighting becomes one-hot. The PGM graphs G1–G4 carry the conceptual load: they make the independence assumption explicit and show both proposed routers arise as conditional expectations under a generative story.

What would settle it

Run the deployed Similarity-Aware and Attention-Aware routers (Definitions 2 and 5, with $\tau = \sigma = 1$) on Wikitext-103 at the final epoch, compute $H(p_i)$ and $H(r_i)$ per token, and count how many tokens — especially low-entropy tokens outside $J_i$ — have strictly higher entropy after mixing; a non-negligible fraction would refute the theoretical justification at the operating point. A second check: shuffle the similarity matrix so each token is weighted by other tokens' scores at random; if routing fluctuation falls as much under shuffled weights as under true similarities, then plain averaging, not the graph structure, is doing the work.

Watch

Extended reading notes

Core claim

Read through a probabilistic graphical model, the (S)MoE generative process draws each token's expert from $\mathrm{Cat}(r(u_i))$ and treats the draws as conditionally independent given the input; the paper identifies this independence, $e_i \perp e_j \mid U$, as the source of routing fluctuation, since nothing forces similar tokens toward the same expert. To break it, the paper places a graph over the tokens and replaces the per-token router score with a mixture $p_i = \sum_j S[i,j] r(u_j)$, where $S$ is either the temperature-softmax similarity matrix $S[i,j] = \mathrm{Softmax}(u_i^\top W_s u_j / \tau)$ (Similarity-Aware) or the posterior attention matrix of the lowest-entropy head (Attention-Aware). Proposition 1 bounds $H(p_i) \leq \sum_j s(i,j) H(r_j) + H(s_i)$ and shows $H(p_i) \leq H(r_i)$ as $\tau \to 0$ or $\sigma \to 0$, giving the theoretical claim that the entropy of expert selection decreases. Empirically, the variants cut the share of tokens switching experts between the last training epochs, lower average routing entropy across layers, improve load balance, and outperform softmax-gated SMoE baselines on Wikitext-103, ImageNet and its corrupted and adversarial variants, and fine-tuning tasks.

Load-bearing premise

Proposition 1's entropy reduction $H(p_i) \leq H(r_i)$ is proved only for tokens whose every neighbor has no larger selection entropy than they do (the set $J_i$), and only in the limits $\tau \to 0$ or $\sigma \to 0$, where the weighting matrix becomes one-hot and no mixing occurs; the deployed Definitions 2 and 5 average over all tokens at $\tau = \sigma = 1$, and the paper assumes without proof that the guarantee survives both relaxations.

Editorial extensions

If this is right

  • Because the change is confined to how router scores are combined, the method layers onto existing SMoE routers; the paper demonstrates this by improving X-MoE, SMoE-dropout, and GLaM-style baselines with the same weighted-mixing trick.
  • If the entropy reduction transfers to training dynamics, late-epoch routing fluctuation — the non-robustness signal the paper measures as tokens switching experts between epochs 59 and 60 — should shrink, and the reported plots show exactly that for both variants.
  • More confident expert selection should make predictions less sensitive to small input perturbations, consistent with the reported gains on adversarially attacked Wikitext-103 and on ImageNet-C, ImageNet-A, and ImageNet-R.
  • Attention-Aware routing reuses a quantity the transformer already computes (the attention matrix), so its overhead is small — reported forward-pass ratios of about 1.07x computation and 1.06x memory over the baseline — making it a cheap upgrade in MoE transformers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: if lower selection entropy is the active ingredient, then any cheap pairwise coupling — for example, averaging router scores within hash-bucketed blocks of tokens — should yield similar stability at far less than the $O(N^2)$ cost of a full similarity matrix, which would matter for very long sequences.
  • A testable extension suggested by the appendix's temperature study: scheduling $\tau$ (and $\sigma$) from large to small during training might preserve the stability gains while sharpening the mixture toward the entropy-reduced limit where Proposition 1 provably holds.
  • The paper notes as its own limitation that it does not model how tokens are generated; a generative model that includes the token-construction step would place the independence problem in the prior rather than patching it at the router, so the present work can be read as evidence for that more principled fix.
  • For deployment, the honest reading is empirical: the proven guarantee holds at one-hot limits, so choosing $\tau = \sigma = 1$ rests on the observed correlation between smoother routing, lower entropy, and fewer switches rather than on Proposition 1 itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes two router modifications for sparse Mixture of Experts: Similarity-Aware (S)MoE, which replaces the per-token router score r(u_i) with a similarity-weighted average sum_j S[i,j] r(u_j) (Def. 2, Eq. 8), and Attention-Aware (S)MoE, which uses a posterior attention matrix A^p_{h*}[i,j] in place of S (Def. 5, Eq. 16). The authors introduce probabilistic graphical models (G1-G4) to motivate these designs, and Proposition 1 claims that the entropy of the resulting expert-selection distribution satisfies H(p_i) <= H(r_i) in the limits tau->0 (similarity) or sigma->0 (attention). Experiments on Wikitext-103, ImageNet, and three downstream tasks report improved perplexity/accuracy, lower routing fluctuation in the final epochs, and lower average routing entropy, with roughly 5% extra compute.

Significance. If the entropy-reduction guarantee held for the implemented algorithm, the contribution would be significant: it offers a nearly drop-in router change that stabilizes late-training SMoE routing, with thorough experiments across language, vision, and fine-tuning settings and with plug-and-play integrations into X-MoE and SMoE-dropout. The paper also includes useful ablations on temperature, number of experts, TopK values, and compute/memory overhead. A strength is the breadth of empirical validation and the clear formulation of the averaging mechanism. However, the advertised theoretical guarantee does not cover the actual algorithm: the proof is restricted to a subset J_i of tokens and to one-hot limits, while the implementation sums over all tokens at tau=sigma=1 and applies TopK and additional approximations. The PGM framework is a modeling choice rather than an independent derivation, but because the paper does not claim to fit the PGM, I do not view that as a correctness error. The significance is therefore conditional on reconciling the theorem with the implementation or reframing the theoretical claim.

major comments (3)
  1. [Section 4 (Prop. 1, Eq. 17) vs. Defs. 2, 5 and Appendix D.1.2] The entropy bound is proved for p_i = sum_{j in J_i} s(i,j) r_j, where J_i = {j : H(e_j|u_j) <= H(e_i|u_i)}, and the inequality H(p_i) <= H(r_i) is obtained only by taking tau->0 or sigma->0 so that H(s_i) -> 0. The implemented models, however, sum over all j = 1..N and are run at tau = sigma = 1 as stated in Appendix D.1.2. At finite temperature H(s_i) is positive, and tokens outside J_i may have H(e_j) > H(e_i), so the chain H(p_i) <= H(s_i) + sum_j s(i,j) H(e_j) <= H(r_i) is not guaranteed. Since the abstract and Section 4 advertise a proof of entropy reduction as the basis for stable routing, the theoretical claim does not currently cover the algorithm as implemented.
  2. [Section 3.2, Eqs. 14-16] The Attention-Aware implementation further deviates from the generative model used in Proposition 1. Equation 14 selects a single head h* by lowest average attention entropy, and Equation 15 replaces the expectation over U in Eq. 13 with the point estimate U = MHA(X). Neither step is part of the sigma->0 argument in Appendix B.2, which concerns the full posterior A^p_h under the generative model. Consequently, the claimed sigma->0 limit does not apply to the actual Def. 5 computation, and the proposition cannot be cited as a guarantee for Eq. 16.
  3. [Section 2.2, Eq. 8 and Section 4, Prop. 1] The implemented methods apply TopK to the averaged scores, whereas Proposition 1 bounds the entropy of the un-thresholded mixture p_i. TopK is a nonlinear operation, and no argument is given that the entropy or the top-k assignment process inherits the bound; additionally, Fig. 3 measures fluctuation as changes in TopK expert assignments, while the proposition concerns score entropy. The manuscript therefore does not rigorously connect the proven statement to the reported fluctuation reduction. A separate top-k analysis, or an explicit statement that the fluctuation results are empirical only, is needed.
minor comments (6)
  1. [Appendix B.2] The proof writes H(e_i) > H(e_j) for j in J_i, but J_i is defined by H(e_j|u_j) <= H(e_i|u_i); the strict inequality should be non-strict so that equal-entropy tokens are not excluded.
  2. [Section 3.2, Eq. 14] The definition of h* should state how ties in the lowest average attention entropy are broken, since this affects reproducibility of the Attention-Aware results.
  3. [Section 5, Fig. 3 and Tables 1-3] No standard errors or number of seeds are reported; this is especially important for Fig. 3, where the central claims of reduced entropy and reduced fluctuation are made.
  4. [Appendix D.1.2] The text says 'selecting Top-1 or Top-2 experts (E = 2)', but E denotes the total number of experts (16) elsewhere; the notation should be corrected to K, the number of selected experts.
  5. [Supplement headings] Several typos remain, including 'Techincal Proofs', 'poterior head selection matrx', 'Comuputation and memory', and 'refered'; a proofreading pass is needed.
  6. [General] The authors state that their implementation builds on publicly available codebases but do not release their own code; providing code would materially strengthen reproducibility.

Circularity Check

1 steps flagged · score 2.0 of 10

PGM 'derivations' of the proposed routers are definitional identities, but the entropy theorem and empirical evaluation are independent of those derivations.

  1. self definitional [Sec. 2.2 (Def. 1-2, Eqs. 5-8); also Sec. 3.2 (Def. 4, Lemma 1, Eqs. 15-16)]
    "si|U∼ Cat(Softmax(u⊤ i WsUT τ )) e s i |si, U∼ Cat (r(usi)) ... ¯oi = E[oi|U] = ... = E X e=1 N X s i=1 re(us i )P(si|U)ge(ui). ... With this result, we now define Similarity-Aware (S)MoE: ¯oi = E X e=1 N X j=1 S[i,j ]re(uj)ge(ui)."

    Def. 1 sets the expert-selection variable e_i^s to be drawn from Cat(r(u_{s_i})), the baseline router evaluated at the similar token. Marginalizing over s_i with weights S[i,j] immediately gives P(e_i^s=e|U)=sum_j S[i,j] r_e(u_j). Eq. (7) then computes E[o_i|U] using that marginal, and Def. 2 renames the result as the Similarity-Aware (S)MoE output. The 'derivation' is thus a definitional identity: the PGM was constructed so its marginal expert-selection distribution equals the proposed routing scores, so recovering the formula via tower rule adds no independent constraint. The same construction occurs in Def. 4/Lemma 1/Eqs. (15)-(16) for Attention-Aware (S)MoE, where e_i^a|z_i,U ~ Cat(r(u_{z_i})) and the final routing score is the posterior over z_i under an attention-based likelihood.

full rationale

No load-bearing circularity. The PGM-to-algorithm steps for Similarity-Aware and Attention-Aware (S)MoE are self-definitional in the sense that the generative models are built so that their posterior/conditional expectations reproduce the proposed routing formulas; this is a modeling choice, not an independent derivation of the routers. However, the paper's central theoretical claim, Proposition 1, is a genuine entropy inequality: it upper-bounds the entropy of a mixture by the weighted average of component entropies plus the weight entropy, and the reduction H(pi) ≤ H(ri) is shown in the stated τ→0/σ→0 limits. That theorem is not obtained by fitting and is not equivalent to its inputs. The gap between the proved restricted-J_i, zero-temperature statement and the implemented all-token, τ=σ=1 setting is a correctness risk, not circularity, because nothing in the proof is assumed from the conclusion. The method introduces no fitted constants: τ and σ are fixed at 1 and ablated (Table 9), W_s is set to identity, and the perplexity/accuracy numbers are reported on standard held-out test sets. Self-citations are limited to codebase references and are not load-bearing. Score 2 reflects the minor self-definitional framing of the PGM 'derivations'; the empirical and theoretical content stands independently.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No constants are fitted to data to obtain the reported results; tau and sigma are hand-set hyperparameters with ablations. The main ledger items are hidden modeling assumptions: the asserted independence-fluctuation link, the single-head heuristic, the point-estimate approximation, and the unproved transfer of the entropy bound to the implemented algorithm.

free parameters (2)
  • tau (similarity temperature) = 1
    Set by hand in all main experiments and ablated in Appendix E.5. The theoretical guarantee requires tau -> 0, which is not the setting used.
  • sigma (attention likelihood variance) = 1
    Set by hand and ablated in Appendix E.5. Same theory-practice mismatch as tau.
assumptions (5)
  • standard math MoE output equals the conditional expectation of the target under a Gaussian expert-output generative model (Eqs. 3-4).
    Used in Sec. 2.1 to read the SMoE formula as an optimal estimate; standard in mixture-of-experts literature (Bishop and Svensen).
  • domain assumption Independence of expert selections across tokens is the cause of routing fluctuation.
    Sec. 2.1 'Limitations' asserts this without derivation; the empirical fluctuation could have other causes, such as learning-rate dynamics or representation shift.
  • ad hoc to paper The attention head with lowest average entropy is the appropriate one for routing (Eq. 14).
    No derivation is given for why h* = argmin E[H(Ah[i,:])] is optimal; it is a heuristic simplification of the full posterior over heads.
  • ad hoc to paper The point estimate U = MHA(X) is a valid replacement for the expectation over U in Eq. 13.
    Used to derive Eq. 15; the paper acknowledges the expectation has no closed form and substitutes the attention output, which is an approximation, not an equality.
  • ad hoc to paper The entropy bound proven for the restricted set Ji transfers to the implemented all-token algorithm at finite temperature.
    Proposition 1 is stated for tokens in Ji and for tau -> 0 or sigma -> 0; Definitions 2 and 5 use all tokens and tau=sigma=1. The paper does not prove the transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Routing in Sparse Mixture of Experts with Graph of Tokens." pith.science (2026). https://pith.science/paper/OIWPZEJE

@misc{pith2026250500792,
  author       = {Pith},
  title        = {Pith review of: Improving Routing in Sparse Mixture of Experts with Graph of Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIWPZEJE}},
  note         = {Machine review of arXiv:2505.00792}
}
read the original abstract

Sparse Mixture of Experts (SMoE) has emerged as a key to achieving unprecedented scalability in deep learning. By activating only a small subset of parameters per sample, SMoE achieves an exponential increase in parameter counts while maintaining a constant computational overhead. However, SMoE models are susceptible to routing fluctuations--changes in the routing of a given input to its target expert--at the late stage of model training, leading to model non-robustness. In this work, we unveil the limitation of SMoE through the perspective of the probabilistic graphical model (PGM). Through this PGM framework, we highlight the independence in the expert-selection of tokens, which exposes the model to routing fluctuation and non-robustness. Alleviating this independence, we propose the novel Similarity-Aware (S)MoE, which considers interactions between tokens during expert selection. We then derive a new PGM underlying an (S)MoE-Attention block, going beyond just a single (S)MoE layer. Leveraging the token similarities captured by the attention matrix, we propose the innovative Attention-Aware (S)MoE, which employs the attention matrix to guide the routing of tokens to appropriate experts in (S)MoE. We theoretically prove that Similarity/Attention-Aware routing help reduce the entropy of expert selection, resulting in more stable token routing mechanisms. We empirically validate our models on various tasks and domains, showing significant improvements in reducing routing fluctuations, enhancing accuracy, and increasing model robustness over the baseline MoE-Transformer with token routing via softmax gating.

Figures

Figures reproduced from arXiv: 2505.00792 by the authors.

Figure 1
Figure 1. PGMs for (S)MoE (G1) and Similarity-Aware (S)MoE (G2). Directed paths are shown by arrows; dotted arrow indi￾cates concatenation; blue arrows highlight differences. U = [u1, . . . , uN ] T is the input sequence of (S)MoE and Similarity￾Aware (S)MoE. ui, oi are input-output at position i ∈ [1, ..., N]. Variables ei and e s i denote expert selection for ui in (S)MoE and Similarity-Aware (S)MoE, respectively; si repres… view at source ↗
Figure 2
Figure 2. PGMs for (S)MoE-Attention (G3) and Attention-Aware (S)MoE (G4) defined in Def. 3 and Def. 4, respectively. Directed paths shown by arrows; dotted arrow indicates concatenation; blue arrows highlight differences. for token ui . MAM generates the output oi as follow: hi ∼ Uniform({1, . . . , H}) zi |hi , X ∼ Cat Softmaxx ⊤ i W⊤ Q,hiWK,hiXT p Dqk  ! ui |zi , hi , X ∼ N (Whixzi , σ2 I) ei |ui ∼ Cat(r(ui)) oi |ui , ei … view at source ↗
Figure 3
Figure 3. Comparison of routing fluctuation and entropy ratio across layers for Baseline SMoE, Attention-Aware SMoE, and Similarity-Aware SMoE and Banking-77 (Casanueva et al., 2020) datasets [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison of Routing Fluctuation and Entropy Ratio Across Layers for Baseline SmoE Top-1, Attention-Aware SMoE Top-1, and Similarity-Aware SMoE Top-1 [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws

    cs.LG 2026-07 accept novelty 6.0 of 10

    A hierarchical copula over Gumbel routing noise changes joint expert choices across tokens while preserving every token's ordered top-K law, mixture weights, and inclusion probabilities.

Reference graph

Works this paper leans on

51 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    V., Du, J., Iyer, S., Pasunuru, R., et al

    Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S., Pasunuru, R., et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684, 2021

    arXiv 2021

  2. [2]

    L., Darrell, T., Malik, J., and Efros, A

    Bai, Y., Geng, X., Mangalam, K., Bar, A., Yuille, A. L., Darrell, T., Malik, J., and Efros, A. A. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22861--22872, 2024

    work page 2024

  3. [3]

    Beit: Bert pre-training of image transformers

    Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  4. [4]

    K., Aggarwal, K., Som, S., Piao, S., and Wei, F

    Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems, 35: 0 32897--32912, 2022

    2022

  5. [5]

    Conditional computation in neural networks for faster models

    Bengio, E., Bacon, P.-L., Pineau, J., and Precup, D. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015

    arXiv 2015

  6. [6]

    and Svensén, M

    Bishop, C. and Svensén, M. Bayesian hierarchical mixtures of experts. In Proceedings Nineteenth Conference on Uncertainty in Artificial Intelligence, pp.\ 57--64. Morgan Kaufmann, January 2003. URL https://www.microsoft.com/en-us/research/publication/bayesian-hierarchical-mixtures-of-experts/

    work page 2003

  7. [7]

    B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  8. [8]

    Efficient intent detection with dual sentence encoders

    Casanueva, I., Temcinas, T., Gerz, D., Henderson, M., and Vulic, I. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020, mar 2020. URL https://arxiv.org/abs/2003.04807. Data available at https://github.com/PolyAI-LDN/task-specific-datasets

    arXiv 2020

Show all 51 references
  1. [9]

    K., Liu, S., and Wang, Z

    Chen, T., Zhang, Z., JAISWAL, A. K., Liu, S., and Wang, Z. Sparse moe as the new dropout: Scaling dense and self-slimmable transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=w1hwFUb_81

    2023

  2. [10]

    On the representation collapse of sparse mixture of experts

    Chi, Z., Dong, L., Huang, S., Dai, D., Ma, S., Patra, B., Singhal, S., Bajaj, P., Song, X., Mao, X.-L., et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35: 0 34600--34613, 2022

    2022

  3. [11]

    Approximating two-layer feedforward networks for efficient transformers

    Csord \'a s, R., Irie, K., and Schmidhuber, J. Approximating two-layer feedforward networks for efficient transformers. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 674--692, Singapore, December 2023...

    2023 doi

  4. [12]

    Stablemoe: Stable routing strategy for mixture of experts

    Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., and Wei, F. Stablemoe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 7085--7095, 2022

    2022

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  6. [14]

    G., Khiem, L., Pham, Q., Nguyen, T., Doan, T.-N., Nguyen, B., Liu, C., Ramasamy, S., Li, X., and Hoi, S

    Do, T. G., Khiem, L., Pham, Q., Nguyen, T., Doan, T.-N., Nguyen, B., Liu, C., Ramasamy, S., Li, X., and Hoi, S. H yper R outer: Towards efficient training and inference of sparse mixture of experts. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Confere...

    2023 doi

  7. [15]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp.\ 5547--5569. PMLR, 2022

    2022

  8. [16]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021

    2021 arXiv

  9. [17]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  10. [18]

    J., Prasad, M., Ramabhadran, B., and Zhu, Y

    Gaur, N., Farris, B., Haghani, P., Leal, I., Moreno, P. J., Prasad, M., Ramabhadran, B., and Zhu, Y. Mixture of informed experts for multilingual speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 623...

    2021

  11. [19]

    Gormley, I. C. and Murphy, T. B. Mixture of experts modelling with social science applications. Mixtures: Estimation and applications, pp.\ 101--121, 2011

    2011

  12. [20]

    The elements of statistical learning: data mining, inference, and prediction, 2009

    Hastie, T. The elements of statistical learning: data mining, inference, and prediction, 2009

    2009

  13. [21]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019

    1903 arXiv

  14. [22]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Compute...

    2021

  15. [23]

    Natural adversarial examples

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15262--15271, 2021 b

    2021

  16. [24]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3 0 (1): 0 79--87, 1991

    1991

  17. [25]

    Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6 0 (2): 0 181--214, 1994

    1994

  18. [26]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  19. [27]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  20. [28]

    Base layers: Simplifying training of large, sparse models

    Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp.\ 6265--6274. PMLR, 2021

    2021

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023

    2023

  22. [30]

    Sparsity-constrained optimal transport

    Liu, T., Puigcerver, J., and Blondel, M. Sparsity-constrained optimal transport. In The Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 10012--10022, 2021

    2021

  24. [32]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843

    2016 arXiv

  25. [33]

    X., Lifland, E., Yoo, J

    Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909, 2020

    2005 arXiv

  26. [34]

    and Dirk, W

    Neil, H. and Dirk, W. Transformers for image recognition at scale. Online: https://ai. googleblog. com/2020/12/transformers-for-image-recognitionat. html, 2020

    2020

  27. [35]

    V., Doan, T

    Nguyen, N. V., Doan, T. T., Tran, L., Nguyen, V., and Pham, Q. Libmoe: A library for comprehensive benchmarking mixture of experts in large language models, 2024. URL https://arxiv.org/abs/2411.00918

    2024 arXiv

  28. [36]

    T., Ramasamy, S., Li, X., Hoi, S., and Ho, N

    Pham, Q., Do, G., Nguyen, H., Nguyen, T., Liu, C., Sartipi, M., Nguyen, B. T., Ramasamy, S., Li, X., Hoi, S., and Ho, N. Competesmoe -- effective training of sparse mixture of experts via competition, 2024

    2024

  29. [37]

    A., and Levy, O

    Press, O., Smith, N. A., and Levy, O. Improving transformer models by reordering their sublayers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 2996--3005, Online, July 2020. Association for Computational Linguistics. doi:10.1...

    2020 doi

  30. [38]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  31. [39]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  32. [40]

    Scaling vision with sparse mixture of experts

    Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34: 0 8583--8595, 2021

    2021

  33. [41]

    Hash layers for large sparse models

    Roller, S., Sukhbaatar, S., Weston, J., et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34: 0 17555--17566, 2021

    2021

  34. [42]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  35. [43]

    D., Ng, A., and Potts, C

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp.\ 1631--1642, Seatt...

    2013

  36. [44]

    Maskmoe: Boosting token-level learning via routing mask in mixture-of-experts

    Su, Z., Lin, Z., Bai, X., Wu, X., Xiong, Y., Lian, H., Ma, G., Chen, H., Ding, G., Zhou, W., et al. Maskmoe: Boosting token-level learning via routing mask in mixture-of-experts. arXiv preprint arXiv:2407.09816, 2024

    2024 arXiv

  37. [45]

    W., and Gholami, A

    Subramanian, S., Harrington, P., Keutzer, K., Bhimji, W., Morozov, D., Mahoney, M. W., and Gholami, A. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems, 36, 2024

    2024

  38. [46]

    Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Open and efficient foundation language models. Preprint at arXiv. https://doi. org/10.48550/arXiv, 2302, 2023

    doi

  39. [47]

    St-moe: Designing stable and transferable sparse expert models

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  41. [49]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  42. [50]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  43. [51]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.