Pith. sign in

REVIEW 4 major objections 4 minor 20 references

SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpecDrop shows category-conditioned routing helps only when each training unit carries one clean label, and ties the no-routing baseline on fuzzy text partitions.

desk verdict Well-executed and transparent paper, but the central claim that routing helps is confounded with label access; the paper's own masking control undermines the attribution. read the letter →

arxiv 2608.04084 v1 pith:UP4RNBS3 submitted 2026-08-04 cs.LG cs.CLcs.CV

Boyao Wang , Zhihan Lei This is my paper · ORCID
classification cs.LGcs.CLcs.CV MSC 68T07
keywords mixture-of-expertscategory-conditionedroutingmodularspecializationgranularityalignmentparameter-freefixed-denominatormergeImageNetBREEDSLoRAinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why learned routers in mixture-of-experts networks often underperform equal-weight No-Routing baselines at matched parameter budgets, and answers that the bottleneck is not the routing algorithm but the alignment between training-signal granularity and the routing target categories. It introduces SpecDrop, a fixed parameter-free scheme in which each of K branches receives a high activation probability for its assigned category and a small leakage otherwise, merged through a category-independent fixed denominator. On aligned vision partitions, where each image carries one clean superclass label, SpecDrop exceeds parameter-matched label-free baselines by +16.15 accuracy on CIFAR-100 and +6.53 on ImageNet-1K BREEDS. On fuzzy language partitions, where training units span multiple categories, the same mechanism ties matched controls within seed noise (+0.10 perplexity, +0.0012 ROUGE-L). The result matters because it locates when category-conditioned routing helps and provides a matched-supervision ceiling against which learned routers can be scored.

What carries the argument

The central object is SpecDrop, a fixed category-conditioned routing rule: a round-robin assignment matrix maps each category to one preferred branch, which receives activation probability $p_{\mathrm{active}}$ while all other branches receive a small leakage $p_{\mathrm{inactive}} > 0$, and the branch outputs are merged through the category-independent fixed denominator $S = p_{\mathrm{active}} + (K-1)p_{\mathrm{inactive}}$, with an optional always-on shared expert added after normalization. The fixed denominator makes the train and test forward passes match exactly and calibrates the merged branch output to single-branch scale, so the shared expert composes co-equally with the routed mixture. The ratio $p_{\mathrm{active}}/p_{\mathrm{inactive}}$ concentrates gradient magnitude on each branch's assigned categories (Theorem 1) while the nonzero leakage preserves cross-category gradient flow, and the HardCategory ablation (one-hot routing, $p_{\mathrm{inactive}}=0$) collapses performance on CIFAR-100, isolating leakage as the active ingredient.

What would settle it

Run the missing No-Routing+SE control on CIFAR-100 at identical branch widths and training budget and compare it against SpecDrop's 79.23%; if the control closes most of the 16.15-point gap, the routing contribution is an artifact of the weak baseline rather than granularity alignment.

Watch

Extended reading notes

Core claim

The paper's central claim is that specialization in modular networks is governed by the alignment between training-signal granularity and the target categories, not by the choice of routing function. Concretely, SpecDrop — a fixed router with zero learned routing parameters and zero auxiliary losses — delivers large gains on partitions where each training unit has one clean category label: 79.23% top-1 on CIFAR-100 (+16.15 over the architecture-matched No-Routing control) and 79.89% top-1 on ImageNet-1K BREEDS (+6.53 over the matched No-Routing+SE control), with branch-category alignment of 58.3% and 100%. On fuzzy partitions (SlimPajama-6B language modeling with 7 document domains; SuperNI instruction tuning over Llama-3.2-1B with LoRA), SpecDrop reduces to the matched No-Routing controls within seed noise (+0.10 PPL, +0.0012 ROUGE-L). The paper further claims that SpecDrop's contribution is converting the label into trained-in modular structure: given the same label at inference, masking a dense model's outputs is stronger for accuracy alone (85.23 on CIFAR, 83.65 on ImageNet), whereas masking SpecDrop's outputs adds 0.00 on CIFAR and +1.06 on ImageNet, meaning the output-space restriction is largely internalized during training.

Load-bearing premise

The load-bearing premise is that the MultiBranch No-Routing control on CIFAR-100 at K=20 (63.08%, with deliberately narrow branch channels [4,7,14]) is a valid mechanism-off baseline; if that multi-branch architecture is pathologically undertrained without routing and no No-Routing+SE control is run on CIFAR, the +16.15 'routing-only' gain is inflated.

Editorial extensions

If this is right

  • On aligned partitions, category-conditioned routing with a fixed, parameter-free schedule beats parameter-matched label-free baselines: +16.15 top-1 on CIFAR-100 and +6.53 on ImageNet-1K BREEDS.
  • On fuzzy partitions, the same mechanism ties the matched No-Routing controls within seed noise (+0.10 PPL on SlimPajama, +0.0012 ROUGE-L on SuperNI), so routing gains are localized by granularity alignment, not by the routing algorithm.
  • The accuracy gains come from trained-in modular structure rather than from label access alone: label-aware logit masking of a dense model reaches 85.23/83.65, while masking SpecDrop changes its score by 0.00 on CIFAR-100 and only +1.06 on ImageNet, and branch-category alignment reaches 58%/100%.
  • The matched-supervision margin provides a ceiling that learned routers can be scored against: the tuned Soft MoE recovers roughly half of the ImageNet ceiling (76.69 vs 73.36 control), while ALF and Mod-Squad remain below the control.
  • If the category label must be predicted, SpecDrop stays above the matched architecture control down to about 80% label accuracy on CIFAR-100 and about 88.5% on ImageNet, but beating the dense reference requires roughly 92-94% label accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not run: on dense prediction tasks such as segmentation, detection, or VQA, whole-input tags should yield no routing gain; if region-level tags are assigned per object, the aligned condition is restored and the vision gains should reappear.
  • A co-trained label predictor could remove the inference-time label requirement; the paper's label-quality curve suggests a predictor that clears roughly 80% category accuracy would keep SpecDrop above its matched control on CIFAR-100.
  • On fuzzy partitions, SpecDrop still produces measurable branch-category alignment (94.4% on SlimPajama) without aggregate metric gains, which implies that routing can reorganize internal structure without changing task performance; future evaluations should report structural metrics alongside task metrics.
  • The fixed-to-learned handover (train under SpecDrop, then distill into a learned label-free router) is a plausible way to import this specialization into standard MoE deployment, since the paper shows the fixed signal builds alignment that a later learned router could inherit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SpecDrop, a fixed, parameter-free category-conditioned routing scheme in which each branch receives activation probability p_a for its assigned category and a small leakage p_i otherwise, merged through a category-independent fixed denominator. The method requires a category label at inference and has no learned routing parameters or auxiliary losses. The authors evaluate SpecDrop on four settings: CIFAR-100 with ResNet-110, ImageNet-1K with a ViT-S/16 on a BREEDS-derived 46-supercategory partition, SlimPajama language modeling with a 30M Transformer, and SuperNI instruction tuning over Llama-3.2-1B with LoRA. They report large gains over No-Routing controls on the two vision settings (+16.15 on CIFAR-100, +6.53 on ImageNet) and null effects on the two NLP settings (+0.10 PPL, +0.0012 ROUGE-L). The abstract's central thesis is that granularity alignment between training-signal granularity and routing target categories, not the routing algorithm, determines when routing helps.

Significance. If the attribution were fully supported, the paper would make a useful contribution: it demonstrates that a fixed, parameter-free routing mechanism can match or exceed learned routers on aligned vision partitions, and it provides an honest masking control showing that output-space restriction is largely internalized during training. The paper is also well controlled in several respects: three seeds per cell, matched-parameter architecture controls, a label-quality curve, a per-category routing extension with a provable denominator invariance, and unusually detailed appendices disclosing training regimes, baseline adaptations, and compute. The central weakness is that the evidence does not cleanly separate the routing mechanism from label access: the headline routing margins are measured against No-Routing controls that do not consume the label, while the paper's own label-aware masking control shows that a non-routing deployment with the same label is at least as accurate. The claim that algorithm choice is not the binding factor is also untested, because no learned router in the comparison receives the category label.

major comments (4)
  1. [Abstract; Table 5; App. E.3, Table 9] The headline routing contributions (+16.15 on CIFAR-100 and +6.53 on ImageNet-1K) are computed against No-Routing and No-Routing+SE controls that do not receive the category label at inference. The paper's own information-matched masking control (App. E.3, Table 9) provides the relevant label-aware comparison: with logit masking, the CIFAR No-Routing control reaches 78.47 versus SpecDrop's 79.23, and the ImageNet No-Routing+SE control reaches 81.44 versus SpecDrop's 80.95. The routing margin over a label-aware, non-routing deployment is therefore +0.76 on CIFAR and -0.49 on ImageNet, so the experiments support the narrower conclusion that category supervision helps on aligned partitions, not that routing itself helps. This directly undermines the abstract's closing thesis that granularity alignment, not algorithm choice, localizes when routing helps.
  2. [Sec. 5.2, Table 2; Sec. 2] The 'not algorithm choice' clause is untested under matched label access. Every learned router evaluated (Soft MoE, ALF, Mod-Squad, COMET) is label-blind, whereas SpecDrop consumes a category label at both training and inference. The advantage over these learned routers is therefore confounded with information asymmetry, not isolated as an algorithmic difference. A learned router that receives the same category signal (for example, a one-hot category embedding concatenated to the router input, or a learned gate conditioned on the label) is needed to support the thesis; without it, the paper can only claim that a fixed label-conditioned schedule beats label-free learned routers at this parameter budget.
  3. [Sec. 5.1, Table 1; App. F.1] The CIFAR-100 routing-isolated comparison uses a No-Routing control at 63.08%, which is 11.4 points below the dense ResNet-110 reference (74.48%) and below every other baseline except HardCategory. The paper attributes this to narrow branch channels [4,7,14] but reports no No-Routing+SE control on CIFAR. Since Table 9 shows that this same 63.08 control reaches 78.47% with inference-time masking, the +16.15 routing share is likely inflated by a poorly conditioned multi-branch architecture rather than by the routing mechanism alone. A CIFAR No-Routing+SE control, or another capacity-matched mechanism-OFF variant, is required to make the routing contribution credible.
  4. [Sec. 5.3, Sec. 5.4, App. E.15/E.12] The assignment of the four settings to the aligned/fuzzy/anti-aligned categories appears to be made in light of the results: SuperNI is called anti-aligned because pruning sensitivity shows 0/15 diagonal hits, and SlimPajama is called fuzzy partly because the PPL tie matches the prediction. To make the thesis falsifiable, the data-side alignment criterion should be stated before the experiments and applied uniformly to all four settings. The embedding diagnostic in App. E.15 currently covers only the SlimPajama-vs-CIFAR modality contrast; it does not provide a pre-specified data-side criterion for the SuperNI partition or for the ImageNet BREEDS partition.
minor comments (4)
  1. [Sec. 3.5, Eq. (5)] The notation S_c for the per-category sum is redundant because the text and Lemma 1 immediately show S_c = S for all c; simplifying to a single symbol S throughout would reduce confusion.
  2. [Table 5] The CIFAR row reports No-Routing+SE as 63.08 with the parenthetical '(no SE)', which is a placeholder rather than a value; the table caption should state explicitly that CIFAR has no shared-expert control, so the routing delta is measured from a different reference than the ImageNet row.
  3. [Abstract] The sentence 'exceeding parameter-matched baselines that do not use the label' applies to the dense baseline and the No-Routing controls, but the later masking control shows that label-aware dense deployments are stronger; the abstract's wording should be adjusted so readers do not over-read the dense comparison as a matched-supervision result.
  4. [Figure 1] Because each panel is normalized independently, the visual intensity of the ViT No-Routing panel is not comparable to the CIFAR panel; the caption notes this, but a shared colorbar or an explicit annotation inside the figure would make the caveat more salient.

Circularity Check

1 steps flagged · score 4.0 of 10

SuperNI is retroactively labeled 'fuzzy' after observing the null, creating a self-definitional loop in the central alignment thesis.

  1. self definitional [Sec. 1 (partition definition) and Sec. 5.4 / Fig. 1 (SuperNI classified as fuzzy)]
    "we call a partition aligned when each training unit carries one clean category label, and fuzzy when training units span multiple categories. ... the SuperNI partition is fuzzy enough that the mechanism fails to engage, a predicted null under our thesis."

    SuperNI training units are assigned exactly one cluster ID via the Domains field (App. B.4: each task's first Domains entry is normalized to a root domain and mapped to a single cluster). By the paper's own Sec. 1 definition, such a partition is aligned, not fuzzy. The paper nevertheless places SuperNI in the fuzzy group and justifies it with post-training Align approx 5% and the null routing gain (+0.0012 ROUGE-L), i.e., the partition label is inferred from the outcome it is used to explain. The 'predicted null' for SuperNI is therefore not a prediction from data-side granularity but a relabeling after the fact.

full rationale

The paper is unusually transparent: it provides an information-matched logit-masking control (App. E.3), a label-quality curve (App. E.4), and explicitly disclaims that SpecDrop beats label-aware deployments for accuracy alone. The CIFAR-100 and ImageNet results, and the SlimPajama null (with independent BGE embedding evidence of chunk-level fuzziness), give the granularity-alignment thesis independent empirical content. The circularity is confined to the SuperNI leg of the four-setting argument: despite each SuperNI training instance carrying exactly one cluster label by the paper's own construction, the partition is called 'fuzzy enough' only after the routing gain is observed to be null and after pruning-sensitivity alignment is measured at chance. That is a self-definitional classification loop, but it does not force the whole derivation because the other three settings stand on independent data-side properties. The paper's central claim is therefore partially circular in one supporting case, but retains independent content overall.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims depend on a small set of validation-selected hyperparameters and on assumptions that the gradient-scaling theory is approximate, that category labels are available at inference, and that the four chosen settings isolate the alignment variable. No new physical or structural entities are introduced.

free parameters (5)
  • p_active / p_inactive (pa, pi) = CIFAR 0.7/0.3; ImageNet 0.6/0.4; SlimPajama 0.6/0.4; SuperNI 0.8/0.2
    Keep probabilities for assigned and unassigned branches; chosen by validation sweep per setting and central to the routing signal strength.
  • shared-expert capacity ratio X = CIFAR 0, ImageNet 2.0, SlimPajama 0.5, SuperNI 1.0
    Capacity of the always-on shared expert relative to routed branches; selected by sweep and affects all main numbers.
  • imbalance amplification exponent beta = ImageNet 1, SlimPajama 4, SuperNI 1, CIFAR not applicable
    Controls per-category gap amplification for rare categories; swept on validation subsets and selected per setting.
  • warmup fraction wr = 1.0 for deployed settings
    Fraction of training over which pa ramps; selected by sweep on CIFAR and fixed at 1.0 elsewhere.
  • BREEDS expansion thresholds T and C = T=60, C=10
    Recursive expansion cutoffs that produce the M=46 ImageNet supercategory partition; chosen by the authors rather than derived from the data.
assumptions (5)
  • domain assumption Gradient norm is independent of the mask given the input (mask-independence).
    Assumption (i) of Theorem 1; idealizes the effect of mask sampling on gradients and is not exactly true in practice.
  • domain assumption At initialization all categories produce equal-magnitude base gradients (category symmetry).
    Assumption (ii) of Theorem 1; once specialization develops the ratio widens, so the theorem is a first-order scaling result.
  • domain assumption Category labels are available at inference and are target-derived on the vision settings.
    SpecDrop requires the category tag at deployment; on CIFAR and BREEDS the tag is a coarsening of the fine label, so the dense comparison is not supervision-controlled.
  • domain assumption Embedding cluster diagnostics (BGE silhouette, DINOv2 silhouette) capture the property that determines routing benefit.
    Used to label SlimPajama as fuzzy and vision as aligned; no proof that these embeddings are the right measure of alignment.
  • domain assumption The 10-epoch training regime on 500M tokens does not bias cross-method rankings.
    The authors disclose the regime and report a 1-epoch sanity check; the ours-versus-control tie holds, but the null may be regime-specific.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization." pith.science (2026). https://pith.science/paper/UP4RNBS3

@misc{pith2026260804084,
  author       = {Pith},
  title        = {Pith review of: SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UP4RNBS3}},
  note         = {Machine review of arXiv:2608.04084}
}
abstract

Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines. Is the bottleneck the routing algorithm, or the alignment between training-signal granularity and the target categories? We probe the question with SpecDrop, a fixed parameter-free routing scheme: each of $K$ branches receives weight $p_a$ for its assigned category and a small leakage $p_i > 0$ otherwise, merged through a category-independent fixed denominator, with no learned routing parameters and no auxiliary losses; the category label is required at inference. On vision tasks where each image has one superclass label (CIFAR-100 on ResNet-110; ImageNet-1K on ViT-S/16), SpecDrop reaches 79.23% on CIFAR-100 and 79.89% on ImageNet-1K, exceeding parameter-matched baselines that do not use the label (+4.75 over dense on CIFAR-100; +6.53 over the No-Routing+SE control on ImageNet-1K). These gains quantify what category supervision buys when deployed through routing -- not an advantage over label-aware deployments of the baselines: given the same label, masking a dense model's outputs is stronger for accuracy alone (85.2 / 83.7). SpecDrop's contribution is converting the label into trained-in modular structure: 58%/100% branch-category alignment, and masking gains of 0.00 (CIFAR) / +1.06 (ImageNet) -- the output-space restriction is largely internalized during training. On fuzzy partitions, where training units span multiple categories (SlimPajama-6B language modeling with a 30M Transformer; SuperNI instruction tuning over Llama-3.2-1B with LoRA), the routing mechanism reduces to the matched No-Routing controls within seed noise, the null our thesis predicts. Granularity alignment, not algorithm choice, localizes when routing helps. Code: https://github.com/Beryex/SpecDrop

Figures

Figures reproduced from arXiv: 2608.04084 by the authors.

Figure 1
Figure 1. Per-branch pruning sensitivity across four settings: alignment quality predicts when category-conditioned routing helps. In each panel, x-axis indexes branches, y-axis categories; cell darkness shows performance drop when branch k is removed for category-c samples. Diag-argmax N/M counts categories whose most-pruning-sensitive branch matches the assigned branch under round-robin (seed 42). The aligned vision partiti… view at source ↗
Figure 2
Figure 2. SpecDrop method overview. Each input’s category tag selects one preferred module via a fixed, pre-set assignment matrix A (round-robin, never updated by gradient descent); the preferred module receives activation probability pa while the rest receive a small leakage pi>0. Routed branches are merged through a category-independent denominator S = pa + (K−1)pi , with an optional always-on shared expert added after norm… view at source ↗
Figure 3
Figure 3. The gradient-concentration mechanism (Thm. 1). Backpropagated gradient magnitude for branch k scales with its activation probability pk(c)/S, so over training each branch accumulates pa/pi more gradient signal from its assigned categories than from unassigned ones, while the nonzero pi preserves cross-category flow. From theory to design. Thm. 1 predicts pactive/pinactive as the specialization-driving gradient ratio… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Hyperparameter ablations across four settings. Row 1: activation probability pa. Row 2: imbalance-amplification exponent β on the imbalanced settings; on balanced CIFAR-100 (equal training images per category), the per-category formula reduces to a scalar for all β, so…
Figure 5
Figure 5. Figure 5: Pruning sensitivity on CIFAR-100 (K=20). Rows: superclasses; cols: branches; cell darkness ∝ ∆k,c (Sec. 5.6); blue outlines (left panel) mark each superclass’s round-robin assigned branch. Left: Soft SpecDrop (diag-argmax 13/20, ratio 1.87×). Right: No-Routing (diag-ar…
Figure 6
Figure 6. Figure 6: ImageNet BREEDS-46 test top-1 across training for the deployed all-blocks Soft MoE, the tuned Soft MoE (canonical second-half placement + lr 5×10−4 ), and Soft SpecDrop. Lines are 3-seed means; shaded bands span seed min/max. Milestones (3-seed means) at epochs 10/25/5…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

  1. [1]

    evolutionary dropout

    = ¯p, so H(Z k) =H b(¯p). Thus I(Z k;C) =H b(¯p)− 1 K Hb(pactive)− K−1 K Hb(pinactive). Applying the identity Hb(¯p)−Hb(p) =D KL(p∥¯p) + (¯p−p) log1−¯p ¯p to each p∈ {pactive, pinactive} and combining, I(Z k;C) = 1 K DKL(pactive∥¯p) +K−1 K DKL(pinactive∥¯p) + log1−¯p ¯p · 1 K (¯p−pactive) + K−1 K (¯p−pinactive) . The bracketed remainder equals ¯p−[1 K pac...

    work page 2017

  2. [2]

    Mini-ablation decomposition (mechanism necessity, β-plateau, SE-monotonic).Phases 5a/5b/5c jointly decompose the contribution of each axis on the balanced BREEDS partition

    at(pa, β, X)=(0.6,1,2.0)confirms the 20%-subset selection transfers to full ImageNet-1K. Mini-ablation decomposition (mechanism necessity, β-plateau, SE-monotonic).Phases 5a/5b/5c jointly decompose the contribution of each axis on the balanced BREEDS partition. Starting from (pa, β, X)=(0.5,1,1)(mechanism-OFF, 42.91 top-1), three findings emerge.(i) Mecha...

    work page arXiv 2021

  3. [4]

    Hash Layers [Roller et al., 2021].Scaled-down N= 8 experts with FFN hidden 192 each (parameter-matched; the original paper’s smallest configuration is N=16)

    is added with weight0.01. Hash Layers [Roller et al., 2021].Scaled-down N= 8 experts with FFN hidden 192 each (parameter-matched; the original paper’s smallest configuration is N=16). Each token is mapped to one expert by a fixed random hash table over the vocabulary; the hash table is drawn once at init from a fixed seed and frozen. No routing parameters...

    work page 2021

  4. [6]

    Commonsense → Concepts and Relations → Social Commonsense

    is anchored to data-modality properties — intra-chunk heterogeneity (56.1% of 512-token chunks span ≥2 BGE clusters at k=7 over the full 9,766-chunk validation set, App. E.15) and silhouette modality asymmetry (BGE smax=0.031 vs DINOv2 CIFAR 0.069) — which are properties of the data and embeddings, independent of training-epoch count. Hyperparameter selec...

    work page 2021

  5. [9]

    Joshua Shunk. Neuron-specific dropout: A deterministic regularization technique to prevent neural networks from overfitting and reduce dependence on large training samples.arXiv preprint arXiv:2201.06938,

    work page Pith review arXiv

  6. [10]

    MaskMoE: Boosting token-level learning via routing mask in mixture-of-experts.arXiv preprint arXiv:2407.09816,

    13 Zhenpeng Su, Zijia Lin, Xue Bai, Xing Wu, Yizhe Xiong, Haoran Lian, Guangyuan Ma, Hui Chen, Guiguang Ding, Wei Zhou, and Songlin Hu. MaskMoE: Boosting token-level learning via routing mask in mixture-of-experts.arXiv preprint arXiv:2407.09816,

    arXiv

  7. [11]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts.arXiv preprint arXiv:2408.15664,

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts.arXiv preprint arXiv:2408.15664,

    arXiv

  8. [12]

    standing committee

    Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, and Zining Zhu. The illusion of specialization: Unveiling the domain-invariant “standing committee” in mixture-of-experts models. arXiv preprint arXiv:2601.03425,

    arXiv

Show all 20 references
  1. [13]

    ST-MoE: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906,

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906,

    arXiv

  2. [14]

    All results support the main-text claim that the category-asymmetricdeterministicsoft signal is the active ingredient; the Bernoulli variant and the specificAordering are not

    14 A Stochastic SpecDrop Formulation and Additional Ablations This appendix contains (i) the Stochastic (Bernoulli) SpecDrop variant against which the canonical Soft variant is benchmarked, (ii) a four-corner mask × denominator ablation at the ResNet-110 / CIFAR-100 scale, and...

    2016

  3. [18]

    own-branch

    builds on this insight, proving that the combination of category conditioning with nonzero cross-category gradient flow (pinactive >0 ) creates provable specialization guarantees that neither structural modularity nor random dropout achieve independently. The interac- tion bet...

    2023

  4. [20]

    no specialization

    ROUGE-L F1 selection metric.We use ROUGE-L F1 per Wang et al.’s Tk-Instruct canonical end-to-end: best.pt selected by argmax ROUGE-L over training epochs (per-epoch generation eval); 27 BEST pa/β/X all argmax on 3-seed mean ROUGE-L; main-table reporting on argmax-ROUGE-L check...

    2024

  5. [1991]

    Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    arXiv

  6. [2009]

    Branch-train-merge: Embarrassingly parallel training of expert language models

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306,

    arXiv

  7. [2017]

    Mixture of cluster-conditional LoRA experts for vision-language instruction tuning.arXiv preprint arXiv:2312.12379,

    Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Xin Jiang, Aoxue Li, Zhenguo Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional LoRA experts for vision-language instruction tuning.arXiv preprint arXiv:2312.12379,

    arXiv

  8. [2021]

    LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. InProceedings of the 62nd Annual Meeting of the Associatio...

    1932

  9. [2022]

    PathNet: Evolution channels gradient descent in super neural networks.arXiv preprint arXiv:1701.08734,

    Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution channels gradient descent in super neural networks.arXiv preprint arXiv:1701.08734,

    arXiv

  10. [2023]

    Modeling task relation- ships in multi-task learning with multi-gate mixture-of-experts

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-experts. InACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1930–1939,

    1930

  11. [2024]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto,

    2009

  12. [2025]

    Tianlong Chen, Zhenyu Zhang, Ajay Jaiswal, Shiwei Liu, and Zhangyang Wang

    doi: 10.1038/s41467-024-55188-9. Tianlong Chen, Zhenyu Zhang, Ajay Jaiswal, Shiwei Liu, and Zhangyang Wang. Sparse MoE as the new dropout: Scaling dense and self-slimmable transformers. InInternational Conference on Learning Representations (ICLR), 2023a. Zitian Chen, Yikang S...

    doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.