{"id":"0863a620-826f-43bc-809a-86e9f45f8ccf","arxiv_id":"2501.13074","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AoE replaces the MoE router with expert self-selection based on internal activation-norm ranking, improving downstream accuracy on 700M-4B parameter language models while keeping throughput near traditional MoE.","lead":"Autonomy-of-Experts (AoE) removes the learned router from Mixture-of-Experts language models and lets each expert decide whether to process a token by ranking its own internal activation norm. The authors pre-train models up to 4B parameters and report better downstream accuracy than traditional MoE at comparable throughput.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AoE's gains are not isolated from the selection rule: without a from-scratch random-selection baseline, the norm-based self-selection mechanism cannot be credited as the cause of improvement.","rationale":"The reader's weakest_assumption names the norm-as-competence proxy as the fragile point, and I agree that this is where the argument is least secure. My stress-test sharpens this into a causal-identification problem: even if the norm correlates with competence in some frozen models, the from-scratch experiments do not isolate the norm ranking as the source of AoE's gains, because AoE changes several things at once (removes the router, factorizes Wg, changes the optimization trajectory through dense pre-computation, and interacts differently with Laux). The paper addresses factorization and router-size confounds (Q4 and Q5), but the missing random-selection control is the decisive gap: without it, the self-selection mechanism is not tested. I also note the motivating evidence is mixed across models (Table 1), which strengthens the need for this control. The reader flagged the absence of a random baseline only in the motivating study and focused on norm validity in from-scratch training; my concern extends this to the from-scratch causal test, so agreement is partial. I do not see a reason to change the reader's CONDITIONAL verdict: the empirical claim is plausible and supported by reasonable ablations, but the mechanism claim requires an additional control experiment before the 'autonomy' interpretation can be taken as established. The proposed random-selection baseline is cheap relative to the 100B-token runs already reported and would settle whether the norm ranking is load-bearing. No formal verification exists, and no independent reproduction of the 4B result is available, so CONDITIONAL remains the appropriate verdict.","tokens_in":19283,"tokens_out":5541,"duration_ms":60092,"concrete_test":"In the 732M setup of Section 4.1.1, train the AoE configuration with dlow=128 and Laux (Config. 8) but replace the L2-norm ranking in Algorithm 2 line 6 with a per-layer fixed random permutation of the 8 experts, applied per token so that each expert has equal marginal selection probability; keep all other settings identical (100B tokens, same optimizer, same seeds). Compare average 8-task accuracy against Config. 8's reported 44.39. If the random-selection variant is within roughly 0.3 points, the norm signal is not load-bearing; if it is more than 1 point worse, norm-based self-selection is doing real work. An additional control ranking by the L2 norm of a frozen random Wdown (never updated) would separate the contribution of learned norm alignment from training dynamics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that ranking experts by the L2 norm of xW_down is both trainable and causally responsible for the observed gains. The paper's ablations rule out pure factorization (Q4, Configs. 2 vs 3) and larger router capacity (Q5, Configs. 2 vs 4) at 732M scale, but they never replace the norm-based ranking in the from-scratch setting with a random or fixed (e.g., hash/round-robin) selection while holding the AoE architecture and auxiliary loss fixed. Without this control, AoE's improvement over MoE (Table 2, e.g., Config. 8 average 44.39 vs Config. 4 average 43.71) could stem from the low-rank factorization's training dynamics, the interaction with Laux, or better load balance, rather than from experts' 'awareness' reflected in activation norms. The motivating study (Table 1) is also inconsistent: xWg preserves 91% of MMLU accuracy on Mixtral (64.23 vs 70.35) but only 38% on Phi-3.5 (29.43 vs 78.20), and no random-selection baseline is reported, so the norm signal is not shown to be competence-revealing. If random selection at 732M matches AoE's average accuracy within noise, the paper's core conceptual contribution (Section 3.1) is unsupported, and the empirical gain would be attributable to architectural changes rather than autonomous self-selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Autonomy-of-Experts (AoE), a Mixture-of-Experts variant in which the router is removed and experts select themselves by ranking the L2 norms of low-dimensional projections x W_down, with only the top-K experts continuing the forward pass. W_g is factorized into W_down W_up to make this self-selection efficient. The authors motivate the method with a preliminary study on frozen pretrained MoE models (Table 1), then train 732M-parameter models on 100B tokens under a range of ablations (Table 2) and a 4B-parameter model (Table 5), reporting that AoE outperforms traditional MoE on eight downstream tasks while achieving 96.8% of MoE throughput (Table 4).","tokens_in":19569,"tokens_out":6373,"duration_ms":67126,"significance":"If the central claim holds, AoE is a worthwhile conceptual contribution: it challenges the default router-based assignment and shows that experts can be selected by an internal signal without a learned router, with close to parity throughput. The paper's strengths include a fairly extensive ablation suite (dlow sweep, load-balancing loss, router-capacity control, top-P and expert-choice variants), a controlled toy experiment in Appendix B, a released code repository, and a clear architectural statement of the low-rank factorization. The main weakness is that the causal role of the norm-based ranking is not isolated from the accompanying architectural changes; the motivating frozen-model study is also inconsistent across models and lacks a random baseline. These gaps leave the central attribution under-supported and require additional experiments or a more cautious claim.","major_comments":[{"comment":"The central claim that experts' 'autonomy'—ranking by the L2 norm of x W_down—is responsible for the downstream gains is not isolated by any from-scratch control. Configurations 3 and 4 rule out low-rank factorization and increased router capacity, respectively, but every AoE configuration still couples the norm-based ranking with the factorized expert architecture and the auxiliary loss. A baseline with the same AoE architecture and Laux but with random or fixed (e.g., hash or round-robin) top-K selection is missing. If such a baseline matches AoE's average accuracy, the improvements in Table 2 would be attributable to the factorized architecture and load dynamics rather than to norm-based self-selection. Please add this control or restrict the claims in the abstract and Section 3.1 accordingly.","section":"§4.1.2, Questions 4–5; Table 2"},{"comment":"The preliminary study does not establish that activation norms are competence-revealing. On Phi-3.5-MoE-instruct, selecting by the norm of xWg retains only 29.43 MMLU accuracy versus 78.20 with the router, and even the best norm node (SiLU(xWg)) retains only 38.03 MMLU, well below the 71% preservation claimed on ARC-C. No random-selection baseline is reported for either model, so it is unclear whether the norm ranking beats a trivial fixed or random top-K choice. The motivating evidence should include such a baseline and should report preservation rates across all tasks, not only the best node/task combination.","section":"§3.1, Table 1"},{"comment":"No seed variance, standard deviations, or significance tests are reported for any downstream accuracy number. The margins that support the central claim are small—for example, Config. 2 averages 43.68 versus Config. 7 at 44.39 in Table 2, and Table 5 reports 48.06 versus 49.80—and in the absence of multiple seeds it is difficult to judge whether these differences are reliable. Please report at least three seeds for the key comparisons (e.g., Configs. 2, 3, 7, 10 and the 4B MoE/AoE pair), or provide confidence intervals.","section":"§4.1.1, §4.2, Tables 2 and 5"}],"minor_comments":[{"comment":"The text says AoE achieves 'up to 97%' of the traditional MoE throughput, but the largest value in Table 4 is 49.79/51.42 = 96.8%; please correct the number.","section":"§4.1.2, Finding 8.1; Table 4"},{"comment":"The setup lists ARC-E, PIQA, SIQA, Winogrande, HellaSwag, MNLI, MRPC, QNLI, QQP, and SST-2, but Table 2 reports only eight tasks and omits MRPC and QQP without explanation; this should be clarified since the reported averages are computed over eight tasks.","section":"§4.1.1, task list; Table 2"},{"comment":"The average Entload values quoted for Figures 3(c) and 3(d) are 2.015 and 2.023, a difference of 0.008; calling this 'superior overall balance' is overstated without an indication of variance across layers or data batches.","section":"§4.1.2, Finding 3.1; Figure 3"},{"comment":"Config. 8 (dlow=128 with Laux) is an outlier in that Laux lowers average accuracy relative to Config. 7 (44.04 versus 44.39), yet the paper attributes this to 'task-specific variations' without reporting per-task or seed-level evidence; a brief explanation or acknowledgment of uncertainty would help.","section":"§4.1.2, Finding 3.3; Table 2"},{"comment":"The caption contains garbled LaTeX remnants (for example, 'Ent!\"#$') and should be cleaned up.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea is interesting, but the missing random-selection control in the from-scratch setting is a load-bearing gap for the paper's central attribution claim. The variance issue compounds this. I recommend a major revision that either supplies the missing control and seed-level statistics or substantially weakens the causal claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real idea — replacing the learned router with a ranking by L2 norm of a low-rank projection (xW_down), then letting the top-K experts continue — and the paper gives it a serious, mostly well-designed ablation campaign. But the paper does not fully isolate the mechanism it claims, and the headline numbers come without error bars. I'd treat the improvement as plausible, not yet robust.\n\nWhat's new: router-free expert selection by activation norm is a genuine departure from Pham et al., which keeps the router and uses norms only as training targets. The low-rank factorization to make the pre-computation affordable is practical and well motivated. The ablation set is unusually thoughtful: they check factorization alone (Config 3), a larger router (Config 4), a dlow sweep, compatibility with Top-P and expert-choice, load-balance statistics, and even add a toy experiment for intuition. Code is available. That's good work.\n\nWhere it soft: the central attribution is underdetermined. Configs 3 and 4 rule out factorization and router capacity, but they never replace the norm ranking with a from-scratch random selection while holding everything else (AoE architecture, Laux) fixed. Without that control, the gain could come from the selection rule being load-balanced or from some interaction with the low-rank dynamics, rather than from norms actually tracking competence. The preliminary study in Table 1 doesn't resolve this: the norm preserves Mixtral well but collapses on Phi-3.5 (29.43 vs 78.20 MMLU), so the signal is not universally competence-revealing. Also, there are no seed variances or error bars anywhere; the average gains are around 0.7 points on small tasks, so single-run differences could be noise. The dlow=128 + Laux outlier (44.04 vs 44.39 without Laux) reinforces that sensitivity. Memory is also up notably (57.9 vs 50.6 GB), which \"comparable efficiency\" glosses over.\n\nFor whom: MoE architecture researchers, especially anyone thinking about routing and expert specialization. The paper deserves a serious referee; I'd ask for the random baseline, multiple seeds, and a more careful discussion of the dlow=128 anomaly before trusting the conclusion. My recommendation: engage, but treat the headline gain as provisional.","headline":"Genuinely new MoE routing idea with unusually thorough ablations, but the missing random-selection baseline and zero error bars leave the central mechanism only partially isolated.","tokens_in":20127,"tokens_out":3537,"would_cite":true,"duration_ms":37412,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-selecting experts outperform router-based MoE in pretrained language models.","keywords":["mixture of experts","expert self-selection","activation norm","low-rank factorization","expert routing","sparse activation","language model pretraining","load balancing"],"falsifier":"On a held-out set, compute, for each token and layer, the norm ranking of all experts and measure the per-expert improvement in output quality (for example, the drop in cross-entropy when that expert is forcibly used). If the bottom-ranked experts perform as well as the top-ranked ones, or if the ranking is uncorrelated with output quality, the central competence-norm claim is disproven.","tokens_in":19086,"feed_emoji":"🧠","tokens_out":5724,"duration_ms":50943,"temperature":0.7,"pith_summary":"The paper proposes replacing the learned router in mixture-of-experts language models with a self-selection mechanism: each expert computes a low-dimensional projection of the input, ranks itself by the L2 norm of that projection, and only the top-K experts proceed. The authors argue that a router decides which experts to use without knowing what those experts can actually do, whereas an expert's own internal activation scale can encode its competence for a given token. They pre-train models from 700M to 4B parameters and report that AoE matches traditional MoE in throughput while improving average accuracy on downstream tasks. If correct, this shifts MoE design away from training a separate gate and toward trusting experts' own signals.","feed_headline":"Self-selecting experts beat router-based MoE","feed_subtitle":"AoE ranks experts by their internal activation norm; at 4B parameters it averages 49.8% vs MoE's 48.1%.","key_machinery":"The operative mechanism is the $\\ell_2$ norm of the low-rank projection $x W_{\\text{down}}$, where $W_{\\text{down}}$ compresses the input into a $d_{\\text{low}}$-dimensional cache shared by all experts. Each AoE expert factorizes its gate weight into $W_{\\text{down}} W_{\\text{up}}$; the cache is computed once per layer for all experts, the norms rank the experts, and only the top-K use the cache to complete the forward pass. The low-rank bottleneck is what makes the pre-computation affordable, and the paper's ablations show the norm signal is most informative when $d_{\\text{low}}$ is about one-third of $d_{\\text{model}}$.","core_discovery":"In router-based MoE, the gate that selects experts is trained separately from the experts it selects, so it can choose an expert that is ill-suited to a token, and the expert may adapt in ways that blur its specialization. AoE removes the router entirely: every expert processes the input up to a low-rank bottleneck ($x W_{\\text{down}}$), the $\\ell_2$ norms of these activations are compared, and only the top-K experts continue. The paper's central claim is that with the selection node explicitly designated during from-scratch pre-training, the activation norm becomes a trainable, reliable measure of an expert's capacity for that input, yielding more confident expert selection, better load balance, and lower training loss than router-based MoE, and better downstream performance at every model size tested, with up to 97% of MoE throughput.","pith_inferences":["A direct diagnostic on frozen AoE checkpoints—correlating per-expert norm ranking with per-expert output quality (for example, the drop in loss when the expert is forcibly used)—would isolate whether the norm is truly a competence signal, separate from the end-task averages.","The paper's own note that AoE's efficiency degrades as the number of experts grows and sparsity increases suggests the 4B-parameter results may not extrapolate to very large expert counts; scaling studies with more experts would test this.","If activation norms encode competence, router predictions on pre-trained models are essentially a learned approximation of this signal, which hints that routers could be condensed from an AoE model or replaced by simpler norm-based gates in dense-MoE hybrids."],"forward_implications":["Router-free selection gives better load balance across experts in most layers, so the auxiliary load-balancing loss becomes less necessary.","Expert specialization becomes more pronounced, and selection confidence rises from shallow to deep layers, matching the intuition that deeper layers handle more abstract roles.","AoE keeps its advantage when the selection rule is switched to Top-P or expert-choice routing, so the benefit is not tied to top-K token-choice.","At the 4B scale, AoE averages 49.80 downstream accuracy against 48.06 for MoE, with larger gains on some tasks, while retaining up to 97% of MoE's training throughput."],"supporting_citations":[{"why":"Introduces the sparse gated MoE layer that the paper's token-choice MoE baseline descends from.","marker":"Shazeer et al., 2017"},{"why":"Extends sparse MoE to large-scale language modeling, forming the standard pipeline AoE compares against.","marker":"Lepikhin et al., 2021"},{"why":"Provides the load-balancing auxiliary loss that the paper adapts for both MoE and AoE baselines.","marker":"Fedus et al., 2022"},{"why":"Supplies the key-value memory interpretation of FFNs that motivates using internal activation scale as a competence signal.","marker":"Geva et al., 2021"},{"why":"Defines expert-choice routing, one of the alternative selection strategies the paper tests AoE against.","marker":"Zhou et al., 2022"},{"why":"Defines Top-P dynamic routing, another strategy used to test AoE's generality.","marker":"Huang et al., 2024"}],"fun_headline_variants":["Experts pick themselves: routerless MoE wins","No router, no problem: self-routing experts win","Activation norms pick experts, boosting MoE quality","Self-selecting experts outperform router MoE","Drop the router: experts choose their own inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the $\\ell_2$ norm of the low-rank projection $x W_{\\text{down}}$ is a trainable and reliable proxy for how well an expert can process a token; if the norm does not encode competence, the top-K choice is arbitrary and AoE loses its advantage over random selection.","fun_headline_variants_meta":{"raw":{"variants":["Experts pick themselves: routerless MoE wins","No router, no problem: self-routing experts win","Activation norms pick experts, boosting MoE quality","Self-selecting experts outperform router MoE","Drop the router: experts choose their own inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1429,"prompt_tokens":930,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":546,"tokens_out":499,"duration_ms":6060,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:27:00.841649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set, compute, for each token and layer, the norm ranking of all experts and measure the per-expert improvement in output quality (for example, the drop in cross-entropy when that expert is forcibly used). If the bottom-ranked experts perform as well as the top-ranked ones, or if the ranking is uncorrelated with output quality, the central competence-norm claim is disproven.","supporting_citations":[],"review_version":1}