Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Autonomy-of-Experts Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Self-selecting experts outperform router-based MoE in pretrained language models.

desk verdict Genuinely new MoE routing idea with unusually thorough ablations, but the missing random-selection baseline and zero error bars leave the central mechanism only partially isolated. read the letter →

arxiv 2501.13074 v2 pith:Y5A2USQP submitted 2025-01-22 cs.CL cs.AIcs.LG

Ang Lv , Ruobing Xie , Yining Qian , Songhao Wu , Xingwu Sun , Zhanhui Kang , Di Wang , Rui Yan This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords mixtureofexpertsexpertself-selectionactivationnormlow-rankfactorizationroutingsparselanguagemodelpretrainingloadbalancing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing the learned router in mixture-of-experts language models with a self-selection mechanism: each expert computes a low-dimensional projection of the input, ranks itself by the L2 norm of that projection, and only the top-K experts proceed. The authors argue that a router decides which experts to use without knowing what those experts can actually do, whereas an expert's own internal activation scale can encode its competence for a given token. They pre-train models from 700M to 4B parameters and report that AoE matches traditional MoE in throughput while improving average accuracy on downstream tasks. If correct, this shifts MoE design away from training a separate gate and toward trusting experts' own signals.

What carries the argument

The operative mechanism is the $\ell_2$ norm of the low-rank projection $x W_{\text{down}}$, where $W_{\text{down}}$ compresses the input into a $d_{\text{low}}$-dimensional cache shared by all experts. Each AoE expert factorizes its gate weight into $W_{\text{down}} W_{\text{up}}$; the cache is computed once per layer for all experts, the norms rank the experts, and only the top-K use the cache to complete the forward pass. The low-rank bottleneck is what makes the pre-computation affordable, and the paper's ablations show the norm signal is most informative when $d_{\text{low}}$ is about one-third of $d_{\text{model}}$.

What would settle it

On a held-out set, compute, for each token and layer, the norm ranking of all experts and measure the per-expert improvement in output quality (for example, the drop in cross-entropy when that expert is forcibly used). If the bottom-ranked experts perform as well as the top-ranked ones, or if the ranking is uncorrelated with output quality, the central competence-norm claim is disproven.

Watch

Extended reading notes

Core claim

In router-based MoE, the gate that selects experts is trained separately from the experts it selects, so it can choose an expert that is ill-suited to a token, and the expert may adapt in ways that blur its specialization. AoE removes the router entirely: every expert processes the input up to a low-rank bottleneck ($x W_{\text{down}}$), the $\ell_2$ norms of these activations are compared, and only the top-K experts continue. The paper's central claim is that with the selection node explicitly designated during from-scratch pre-training, the activation norm becomes a trainable, reliable measure of an expert's capacity for that input, yielding more confident expert selection, better load balance, and lower training loss than router-based MoE, and better downstream performance at every model size tested, with up to 97% of MoE throughput.

Load-bearing premise

The method assumes that the $\ell_2$ norm of the low-rank projection $x W_{\text{down}}$ is a trainable and reliable proxy for how well an expert can process a token; if the norm does not encode competence, the top-K choice is arbitrary and AoE loses its advantage over random selection.

Editorial extensions

If this is right

  • Router-free selection gives better load balance across experts in most layers, so the auxiliary load-balancing loss becomes less necessary.
  • Expert specialization becomes more pronounced, and selection confidence rises from shallow to deep layers, matching the intuition that deeper layers handle more abstract roles.
  • AoE keeps its advantage when the selection rule is switched to Top-P or expert-choice routing, so the benefit is not tied to top-K token-choice.
  • At the 4B scale, AoE averages 49.80 downstream accuracy against 48.06 for MoE, with larger gains on some tasks, while retaining up to 97% of MoE's training throughput.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct diagnostic on frozen AoE checkpoints—correlating per-expert norm ranking with per-expert output quality (for example, the drop in loss when the expert is forcibly used)—would isolate whether the norm is truly a competence signal, separate from the end-task averages.
  • The paper's own note that AoE's efficiency degrades as the number of experts grows and sparsity increases suggests the 4B-parameter results may not extrapolate to very large expert counts; scaling studies with more experts would test this.
  • If activation norms encode competence, router predictions on pre-trained models are essentially a learned approximation of this signal, which hints that routers could be condensed from an AoE model or replaced by simpler norm-based gates in dense-MoE hybrids.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Autonomy-of-Experts (AoE), a Mixture-of-Experts variant in which the router is removed and experts select themselves by ranking the L2 norms of low-dimensional projections x W_down, with only the top-K experts continuing the forward pass. W_g is factorized into W_down W_up to make this self-selection efficient. The authors motivate the method with a preliminary study on frozen pretrained MoE models (Table 1), then train 732M-parameter models on 100B tokens under a range of ablations (Table 2) and a 4B-parameter model (Table 5), reporting that AoE outperforms traditional MoE on eight downstream tasks while achieving 96.8% of MoE throughput (Table 4).

Significance. If the central claim holds, AoE is a worthwhile conceptual contribution: it challenges the default router-based assignment and shows that experts can be selected by an internal signal without a learned router, with close to parity throughput. The paper's strengths include a fairly extensive ablation suite (dlow sweep, load-balancing loss, router-capacity control, top-P and expert-choice variants), a controlled toy experiment in Appendix B, a released code repository, and a clear architectural statement of the low-rank factorization. The main weakness is that the causal role of the norm-based ranking is not isolated from the accompanying architectural changes; the motivating frozen-model study is also inconsistent across models and lacks a random baseline. These gaps leave the central attribution under-supported and require additional experiments or a more cautious claim.

major comments (3)
  1. [§4.1.2, Questions 4–5; Table 2] The central claim that experts' 'autonomy'—ranking by the L2 norm of x W_down—is responsible for the downstream gains is not isolated by any from-scratch control. Configurations 3 and 4 rule out low-rank factorization and increased router capacity, respectively, but every AoE configuration still couples the norm-based ranking with the factorized expert architecture and the auxiliary loss. A baseline with the same AoE architecture and Laux but with random or fixed (e.g., hash or round-robin) top-K selection is missing. If such a baseline matches AoE's average accuracy, the improvements in Table 2 would be attributable to the factorized architecture and load dynamics rather than to norm-based self-selection. Please add this control or restrict the claims in the abstract and Section 3.1 accordingly.
  2. [§3.1, Table 1] The preliminary study does not establish that activation norms are competence-revealing. On Phi-3.5-MoE-instruct, selecting by the norm of xWg retains only 29.43 MMLU accuracy versus 78.20 with the router, and even the best norm node (SiLU(xWg)) retains only 38.03 MMLU, well below the 71% preservation claimed on ARC-C. No random-selection baseline is reported for either model, so it is unclear whether the norm ranking beats a trivial fixed or random top-K choice. The motivating evidence should include such a baseline and should report preservation rates across all tasks, not only the best node/task combination.
  3. [§4.1.1, §4.2, Tables 2 and 5] No seed variance, standard deviations, or significance tests are reported for any downstream accuracy number. The margins that support the central claim are small—for example, Config. 2 averages 43.68 versus Config. 7 at 44.39 in Table 2, and Table 5 reports 48.06 versus 49.80—and in the absence of multiple seeds it is difficult to judge whether these differences are reliable. Please report at least three seeds for the key comparisons (e.g., Configs. 2, 3, 7, 10 and the 4B MoE/AoE pair), or provide confidence intervals.
minor comments (5)
  1. [§4.1.2, Finding 8.1; Table 4] The text says AoE achieves 'up to 97%' of the traditional MoE throughput, but the largest value in Table 4 is 49.79/51.42 = 96.8%; please correct the number.
  2. [§4.1.1, task list; Table 2] The setup lists ARC-E, PIQA, SIQA, Winogrande, HellaSwag, MNLI, MRPC, QNLI, QQP, and SST-2, but Table 2 reports only eight tasks and omits MRPC and QQP without explanation; this should be clarified since the reported averages are computed over eight tasks.
  3. [§4.1.2, Finding 3.1; Figure 3] The average Entload values quoted for Figures 3(c) and 3(d) are 2.015 and 2.023, a difference of 0.008; calling this 'superior overall balance' is overstated without an indication of variance across layers or data batches.
  4. [§4.1.2, Finding 3.3; Table 2] Config. 8 (dlow=128 with Laux) is an outlier in that Laux lowers average accuracy relative to Config. 7 (44.04 versus 44.39), yet the paper attributes this to 'task-specific variations' without reporting per-task or seed-level evidence; a brief explanation or acknowledgment of uncertainty would help.
  5. [Figure 3 caption] The caption contains garbled LaTeX remnants (for example, 'Ent!"#$') and should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the AoE comparison against MoE is external, the norm-based selection is an architectural choice, and the auxiliary claims are descriptive rather than derived from the method's definition.

full rationale

No significant circularity identified. The paper's central empirical claim—that AoE outperforms MoE on downstream tasks (Table 5)—is validated against external benchmarks and standard MoE baselines; it does not reduce to the definition of the selection signal. The norm-based ranking is an architectural design choice, and the preliminary frozen-model study (Table 1) is an independent, parameter-free check against router performance, not a fitted prediction. Ablations (Configs. 3 vs 2, 4 vs 2) explicitly separate the low-rank factorization and larger router capacity from the norm-selection mechanism. The 'awareness' framing is interpretive language applied to the norm, and statements about alignment and confidence are descriptive of the trained model rather than derived from the method's definition. Self-citations (e.g., HMoE, Lv et al. 2024) are peripheral and not load-bearing. The absence of a random-selection baseline in the motivating study is an evidentiary gap, but it does not constitute a circular step under the specified criteria.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its free parameters are standard hyperparameters, and its axioms are domain assumptions about neural network representations and low-rank structure.

free parameters (2)
  • dlow = 256 for 732M model, 400 for 4B model
    The low-rank projection dimension is a design choice varied in Section 4.1 (64, 128, 256, 512) and set to 256 (approximately dmodel/3) for small models and 400 for the 4B model based on accuracy/efficiency trade-offs.
  • alpha_aux = 0.01
    The auxiliary load-balancing loss weight is set after validation on 5 billion tokens from OpenWebText; used for both MoE and AoE models (Section 4.1.2, Question 3).
assumptions (3)
  • domain assumption FFN weights are inherently low-rank, so factorizing W_g into W_down W_up does not impair expressiveness.
    Invoked in Section 3.2 to justify the low-rank factorization, citing Li et al., Aghajanyan et al., and Hu et al. The ablation (Config 3 vs 2) empirically supports this for the small model.
  • domain assumption The scale of an expert's internal activation reflects its ability to process a token.
    Central premise introduced in Sections 1 and 3.1, motivated by Geva et al.'s key-value memory analogy and preliminary experiments. The authors call the analogy 'purely analogical' and offer no formal model.
  • domain assumption The low-dimensional projection x W_down preserves enough information for expert ranking.
    Assumed when selecting the pause node in Algorithm 2. The dlow sweep (Question 2) shows performance degrades at small dlow, consistent with a lossy projection, but the threshold is empirical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Autonomy-of-Experts Models." pith.science (2026). https://pith.science/paper/Y5A2USQP

@misc{pith2026250113074,
  author       = {Pith},
  title        = {Pith review of: Autonomy-of-Experts Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5A2USQP}},
  note         = {Machine review of arXiv:2501.13074}
}
read the original abstract

Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's decision-making and the experts' execution is a critical yet overlooked issue, leading to suboptimal expert selection and ineffective learning. To address this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which experts autonomously select themselves to process inputs. AoE is based on the insight that an expert is aware of its own capacity to effectively process a token, an awareness reflected in the scale of its internal activations. In AoE, routers are removed; instead, experts pre-compute internal activations for inputs and are ranked based on their activation norms. Only the top-ranking experts proceed with the forward pass, while the others abort. The overhead of pre-computing activations is reduced through a low-rank weight factorization. This self-evaluating-then-partner-comparing approach ensures improved expert selection and effective learning. We pre-train language models having 700M up to 4B parameters, demonstrating that AoE outperforms traditional MoE models with comparable efficiency.

Figures

Figures reproduced from arXiv: 2501.13074 by the authors.

Figure 1
Figure 1. Comparison between traditional MoE and AoE. Arrows indicate data flow, while shadowed modules represent unused pa￾rameters or variables. (a) Traditional MoE models use a router to assign tokens to specific experts. This separation between the router‘s decision-making and the experts’ execution leads to sub￾optimal expert selection and ineffective learning. (b) In an AoE model, experts operate autonomously. They are … view at source ↗
Figure 2
Figure 2. Pre-training NLL losses. All configurations shown are trained with Laux, though its value is not included in the figure. 4.1.2. RESOLVING QUESTIONS REGARDING AOE We investigate the following questions related to AoE through a series of ablation experiments. Question 1: How does the downstream performance of AoE compare with traditional MoE models? We evalu￾ated various configurations of AoE (Configs. 5 to 12 ) and t… view at source ↗
Figure 3
Figure 3. Statistical analysis of expert load. The figure reveals several key insights: (1) Laux enhances load balancing in both traditional MoE and AoE. (2) AoEs generally exhibit more balanced load distributions compared to their traditional MoE counterparts, as indicated by higher Entload values. (3) AoEs also demonstrate greater confidence in expert selection, reflected by lower Entconf values. tion. All of these variants… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Average activation norm dynamics during training. Each plot represents an expert, distinguished by color according to its layer. Experts within the same layer achieve similar activation scales, indicating that their self-evaluation criteria for determining whether they…
Figure 5
Figure 5. Figure 5: The overview of our toy experiments training tiny AoE and traditional MoE classifiers. of the idle expert, which lacks specialization and does not resist class three inputs. As a result, the idle expert naturally handles all class three inputs. This results in heteroge…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [6]

    Li, C., Farkhoor, H., Liu, R., and Yosinski, J

    URL https://proceedings.mlr.press/ v139/lewis21a.html. Li, C., Farkhoor, H., Liu, R., and Yosinski, J. Measur- ing the intrinsic dimension of objective landscapes. In International Conference on Learning Representations,

    work page

  2. [9]

    Sakaguchi, K., Bras, R

    URL https://openreview.net/forum? id=lMgDDWb1ULW. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/ 1907.10641. Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y . Social IQa: Commonsense reasoning about social interactions. In Inui, K., Jiang, ...

    arXiv 2019

  3. [12]

    URL https://qwenlm.github.io/blog/ qwen-moe/. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi `ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https://arxiv.org/ abs/2302.13971. Tow, J., Bellagente, M., Ma...

    arXiv 2023

  4. [15]

    Zhou, Y ., Du, N., Huang, Y ., Peng, D., Lan, C., Huang, D., Shakeri, S., So, D., Dai, A

    URL https://openreview.net/forum? id=jdJo1HIVinI. Zhou, Y ., Du, N., Huang, Y ., Peng, D., Lan, C., Huang, D., Shakeri, S., So, D., Dai, A. M., Lu, Y ., Chen, Z., Le, Q. V ., Cui, C., Laudon, J., and Dean, J. Brainformers: Trading simplicity for efficiency. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceed...

    work page

  5. [16]

    the separation between the router’s decision and the experts’ execution

    URL https://proceedings.mlr.press/ v202/zhou23c.html. Zuo, S., Liu, X., Jiao, J., Kim, Y . J., Hassan, H., Zhang, R., Gao, J., and Zhao, T. Taming sparsely activated trans- former with stochastic experts. InInternational Confer- ence on Learning Representations, 2022. URL https: //openreview.net/forum?id=B72HXs80q4. 13 Autonomy-of-Experts Models A. Re-run...

    work page 2022

  6. [407]

    naacl-main.407

    URL https://aclanthology.org/2022. naacl-main.407. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. URL https:// openreview.net/forum?id=d7KBjmI3GmQ. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, ...

    arXiv 2022

  7. [568]

    acl-long.568

    URL https://aclanthology.org/2021. acl-long.568. Bisk, Y ., Zellers, R., Le bras, R., Gao, J., and Choi, Y . Piqa: Reasoning about physical commonsense in natural lan- guage.Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, Apr. 2020. doi: 10.1609/ aaai.v34i05.6239. URL https://ojs.aaai.org/ index.php/AAAI/article/view/6239....

    2021

  8. [1164]

    emnlp-main.1164

    URL https://aclanthology.org/2024. emnlp-main.1164. Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., and Zettlemoyer, L. DEMix layers: Disentangling do- mains for modular language modeling. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V . (eds.),Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Comput...

    2024

Show all 16 references
  1. [2013]

    URL https://aclanthology.org/D13-1170

    Association for Computational Linguistics. URL https://aclanthology.org/D13-1170. Sun, X., Chen, Y ., Huang, Y ., Xie, R., Zhu, J., Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., Bu, J., Chen, Z., Huang, X., Lian, F., Yang, S., Yan, J., Zeng, Y ., Ren, X., Yu, C., Wu, L., Mao,...

    2024 arXiv

  2. [2017]

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C

    URL https://openreview.net/forum? id=B1ckMDqlg. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., and Beth...

    2013

  3. [2018]

    Lin, H., Lv, A., Chen, Y ., Zhu, C., Song, Y ., Zhu, H., and Yan, R

    URL https://openreview.net/forum? id=ryup8-WCW. Lin, H., Lv, A., Chen, Y ., Zhu, C., Song, Y ., Zhu, H., and Yan, R. Mixture of in-context experts enhance LLMs’ long context awareness. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems,

  4. [2019]

    Wang, A., Sun, X., Xie, R., Li, S., Zhu, J., Yang, Z., Zhao, P., Han, J

    URL https://openreview.net/forum? id=rJ4km2R5t7. Wang, A., Sun, X., Xie, R., Li, S., Zhu, J., Yang, Z., Zhao, P., Han, J. N., Kang, Z., Wang, D., Okazaki, N., and zhong Xu, C. Hmoe: Heterogeneous mixture of experts for language modeling, 2024a. URL https://arxiv. org/abs/2408....

    arXiv

  5. [2021]

    doi: 10.18653/v1/2021.emnlp-main.446

    Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.446. URL https:// aclanthology.org/2021.emnlp-main.446. Gokaslan, A. and Cohen, V . Openwebtext cor- pus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. Gong, Z., Lv, A., Guan, J., Wu, W., Zhang...

    2021 doi

  6. [2022]

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O

    URL https://proceedings.mlr.press/ v162/clark22a.html. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/ 1803.05457. Compu...

    2018 arXiv

  7. [2023]

    Wang, L., Gao, H., Zhao, C., Sun, X., and Dai, D

    URL https://openreview.net/forum? id=NpsVSN6o4ul. Wang, L., Gao, H., Zhao, C., Sun, X., and Dai, D. Auxiliary-loss-free load balancing strategy for mixture-of- experts, 2024b. URL https://arxiv.org/abs/ 2408.15664. Williams, A., Nangia, N., and Bowman, S. A broad- coverage cha...

    2018 arXiv

  8. [2024]

    Loshchilov, I

    URL https://openreview.net/forum? id=RcPHbofiCN. Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. InInternational Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Bkg6RiCqY7. 11 Autonomy-of-Experts Models Lv, A., Chen, Y ...

    2019 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.