Pith. sign in

REVIEW 1 major objections 1 minor 45 references

Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks

T0 review · 1 major / 1 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A post-training extraction method turns pretrained vision transformers into mixture-of-experts models, recovering 98% of ImageNet accuracy with up to 36% fewer operations.

desk verdict The headline MACs savings in Table 1 don't survive arithmetic: DeiT-B can't cut 36.3% MACs if only layers 6–11 are converted; the ceiling is 31.7%. read the letter →

arxiv 2505.15414 v1 pith:VYWPMZZE submitted 2025-05-21 cs.CV

Uranik Berisha , Jens Mehnert , Alexandru Paul Condurache This is my paper
classification cs.CV
keywords mixtureofexpertsvisiontransformersexpertextractionactivationclusteringHDBSCANmodelcompressionconditionalcomputationImageNet-1k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a pretrained vision transformer can be converted into a mixture-of-experts model after training, without retraining from scratch. The proposed method clusters the hidden activations of each MLP layer with HDBSCAN, treats each cluster as evidence of an expert subnetwork, and selects that expert's neurons by their within-cluster variance. At inference, tokens are routed by cosine similarity to the mean input vector of each cluster, so only the selected expert's weights are computed. The paper reports that, after a short distillation fine-tune, the resulting MoEE variants of DeiT-Tiny, DeiT-Small, and DeiT-Base retain roughly 98% of baseline top-1 accuracy on ImageNet-1k while cutting multiply-accumulate operations by 27-36% and parameters by 20-32%. If true, this turns existing pretrained models into cheaper conditional-computation models with only a light fine-tuning step.

What carries the argument

Two mechanisms carry the argument. First, the cluster-mean router: each expert is represented by the mean input vector $\vec{\mu}_c$ of the tokens whose activations fall in its cluster, and inference routes a token to the expert maximizing cosine similarity to that mean, replacing density estimation with a single small matrix multiplication. Second, variance-based neuron selection: within each cluster, hidden neurons are ranked by activation variance, and the top $p\%$ of cumulative variance selects the expert's columns in the first linear layer $W^{(1)}$ and the corresponding rows of the second linear layer $W^{(2)}$. HDBSCAN supplies the data-driven cluster count, so layers without detected clusters are left unmodified while deeper layers are split into experts.

What would settle it

On a held-out token set, compare the expert chosen by the input-space cosine router with the expert assigned by the activation-space cluster labels used during extraction. If the agreement is low, then the routing proxy, not the extracted experts, would be the thing to blame for any accuracy loss; a second check is to replace the learned mean input vectors with random fixed vectors and measure how much of the 80.12% top-1 accuracy survives the same fine-tuning procedure.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that expert structures are already present in the MLP layers of trained vision transformers and can be extracted post-training from activation statistics alone. In the proposed MoEE pipeline, HDBSCAN groups token activations in each hidden layer into clusters; because linear maps preserve clusters, each activation cluster is mapped back to the input tokens that produced it, and the cluster is represented by their mean vector $\vec{\mu}_c$. The expert for cluster $c$ is then the set of hidden neurons whose activations have the highest variance inside that cluster, selected until the cumulative variance reaches an extraction percentage $p$, giving the columns of $W^{(1)}$ and the matching rows of $W^{(2)}$. At inference a token goes to the expert with the largest cosine similarity between the token and $\vec{\mu}_c$. The paper claims this recovers 98% of the original DeiT-B top-1 accuracy after 30 epochs of distillation fine-tuning (80.12% vs 81.73%) while reducing MACs by 36.3% and parameters by 32.4%.

Load-bearing premise

The load-bearing premise is that clusters discovered in the hidden activation space can be represented by the mean of the corresponding input tokens, and that cosine similarity to those input-space means routes new tokens to roughly the same experts that the activation clusters would pick.

Editorial extensions

If this is right

  • Pretrained ViTs can be converted into MoE variants with a single extraction pass and a short fine-tune, avoiding the load-balancing losses and from-scratch training of conventional MoEs.
  • Savings grow with model size: DeiT-Base shows the largest reductions (36.3% MACs, 32.4% parameters) and the highest accuracy retention before fine-tuning, so the method is most attractive for larger models.
  • The method extends beyond DeiT: the paper reports above-98% recovered accuracy with 30-45% parameter reductions on Swin and ConvNeXt base variants, and on CIFAR-100-trained DeiTs, suggesting it does not depend on a specific inductive bias.
  • Because experts only form in deeper layers, shallow general-feature layers are preserved unchanged, which keeps the representational structure of early processing intact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the input-space cosine router is a proxy for the activation clusters; the paper ablates it against random routing but not against routing directly in activation space, so a head-to-head comparison would isolate how much of the 80.12% result depends on that proxy.
  • Editorial extension: the fact that only about 640 images (126k tokens) stabilize the extracted expert configurations suggests the method could be applied in low-data or private-data settings, where a full training set is unavailable for the extraction pass.
  • Editorial extension: since extracted experts overlap and unselected neurons are dropped, the method could be composed with structured pruning of the remaining weights to push the MACs and parameter savings further, a direction the paper mentions only as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper proposes a method to convert pretrained Vision Transformers into Mixture-of-Experts variants post-hoc, without training from scratch. The method clusters hidden MLP activations using HDBSCAN, extracts sub-networks from each cluster based on activation variance, and routes tokens during inference via cosine similarity to input-space cluster means. On ImageNet-1k, for DeiT-T/S/B the authors report MACs reductions of 27–36%, parameter reductions of 20–32%, and final Top-1 accuracies that recover 96–98% of the baseline after fine-tuning. Ablations show that each component (clustering, variance-based extraction, input-based routing) contributes positively, and additional experiments on Swin, ConvNeXt, and CIFAR-100 indicate generality.

Significance. If the quantitative claims are correct, the contribution is practically valuable: it offers a data-driven way to obtain MoE-like efficiency from already-trained vision models, avoiding the cost of MoE training from scratch, and it provides insights into emergent modularity in ViTs. The paper includes several genuine strengths: a data-driven expert-count mechanism (HDBSCAN), a variance-based extraction criterion, an ablation study with randomized baselines (Section 5, Table 2), a stability analysis with multiple seeds and standard deviations (Section 9, Table 3), and generalization experiments to other architectures and datasets (Section 11). However, the central MACs-reduction figures are internally inconsistent with the stated layer conversion configuration; this erodes confidence in the headline efficiency claims until the discrepancy is resolved.

major comments (1)
  1. The main quantitative results (Table 1) are reported without error bars or multiple seeds, and the hyperparameters (minimum cluster size 0.6%, extraction percentage 80%) are selected on DeiT-S on ImageNet-1k and then applied to the other models on the same dataset. The paper does include a multi-seed stability experiment in Section 9 (Table 3), but that experiment measures only the effect of sample size on the extraction, not the variance of the final fine-tuned accuracy. To support the claimed accuracy–efficiency trade-offs, the authors should report mean and standard deviation over at least two or three independent extraction and fine-tuning runs for the main configurations, and clarify whether any hyperparameter selection was performed on the same evaluation set.
minor comments (1)
  1. [Figure 9] The labels '80% Variance' and '80% Mean' are unclear; it would be clearer to say 'cumulative variance reaches 80%' and 'cumulative mean reaches 80%'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; method is self-contained, with minor non-load-bearing self-citations.

full rationale

The paper's central claim is that pretrained ViTs can be converted into MoE variants by clustering activation patterns, extracting variance-selected subnetworks, and routing via input-space means. The extraction is unsupervised, the routing is evaluated on held-out validation images, and the final Top-1 accuracy is an external benchmark against the original DeiT models. No predicted quantity is a fitted parameter by construction. The self-citations (e.g., [1] and [22] in the related work) are not load-bearing; the central premise of sparse activation patterns in pretrained transformers is cited to Zhang et al. [32] and [33], not to the authors' own work. The simplifications, such as spherical clusters and input-space routing, are explicit assumptions and are ablated. A separate arithmetic inconsistency exists in the claimed MACs reductions (the reduction exceeds the maximum possible given the stated number of converted layers), but this is an internal consistency issue, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central efficiency claim rests on hand-chosen hyperparameters (cluster size, extraction percentage, sample count) and on the heuristic that input-space centroids can route tokens to the right activation-level experts. There is no formal proof of this routing proxy.

free parameters (4)
  • minimum cluster size percentage = 0.6% of extracted tokens (about 756 tokens for 126,080 samples)
    Controls the number of experts via HDBSCAN; selected by a hyperparameter search on DeiT-S/ImageNet balancing accuracy and MACs (Section 4.1, Appendix 8).
  • extraction percentage p = 80%
    Determines how many hidden neurons each expert keeps by cumulative variance; chosen from a heatmap search (Section 4.1, Appendix 8).
  • number of input images for extraction = 640 images / 126,080 tokens
    Chosen to keep standard deviation below 1% while limiting HDBSCAN runtime (Appendix 9).
  • fine-tuning epochs = 30
    Training budget needed to regain accuracy; affects the efficiency claim (Section 4).
assumptions (4)
  • domain assumption Pretrained ViT MLPs contain modular, sparse subnetworks that can be identified from activation clusters
    Based on prior work on emergent modularity [32,33], assumed to transfer from language to vision (Introduction).
  • domain assumption Clusters found in hidden activation space can be represented by mean vectors of the corresponding input tokens
    The routing uses cosine similarity to input-space cluster means; the paper calls this a simplifying spherical-cluster assumption (Section 3.2.2).
  • domain assumption A random subset of about 126k tokens adequately represents the activation patterns
    Stated in Section 3.1 and tested in Section 5.4.
  • domain assumption Knowledge distillation from the original model during fine-tuning can recover most accuracy
    Used in all experiments; no proof, but empirically validated (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks." pith.science (2026). https://pith.science/paper/VYWPMZZE

@misc{pith2026250515414,
  author       = {Pith},
  title        = {Pith review of: Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VYWPMZZE}},
  note         = {Machine review of arXiv:2505.15414}
}
read the original abstract

Vision Transformers have emerged as the state-of-the-art models in various Computer Vision tasks, but their high computational and resource demands pose significant challenges. While Mixture-of-Experts (MoE) can make these models more efficient, they often require costly retraining or even training from scratch. Recent developments aim to reduce these computational costs by leveraging pretrained networks. These have been shown to produce sparse activation patterns in the Multi-Layer Perceptrons (MLPs) of the encoder blocks, allowing for conditional activation of only relevant subnetworks for each sample. Building on this idea, we propose a new method to construct MoE variants from pretrained models. Our approach extracts expert subnetworks from the model's MLP layers post-training in two phases. First, we cluster output activations to identify distinct activation patterns. In the second phase, we use these clusters to extract the corresponding subnetworks responsible for producing them. On ImageNet-1k recognition tasks, we demonstrate that these extracted experts can perform surprisingly well out of the box and require only minimal fine-tuning to regain 98% of the original performance, all while reducing MACs and model size, by up to 36% and 32% respectively.

Figures

Figures reproduced from arXiv: 2505.15414 by the authors.

Figure 1
Figure 1. Illustration of the expert extraction process. The process begins with the clustering of activations (a), followed by the extraction [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Sorted routing distributions across experts for all ImageNet-1k classes at different layers, demonstrating a more balanced routing [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Token routing distributions at different layers for randomly selected classes (goldfish, pug, plane, and cliff) compared to the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Token routing distributions at different layers for visually similar truck-like classes (fire engine, garbage truck, pickup, and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Similarity matrices of mean expert inputs at different [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Heatmaps as a function of extraction percentage and [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Comparison of token routing distributions for visually similar classes in layer 11. (a) presents routing patterns for truck-like [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Distribution of variance and mean across neurons in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Figure of selected sample patches from DeiT-B [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 38 canonical work pages

  1. [1]

    Cnn mixture- of-depths

    Rinor Cakaj, Jens Mehnert, and Bin Yang. Cnn mixture- of-depths. In Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings, Part VII , page 148–166, Berlin, Heidelberg, 2024. Springer-Verlag. 2

    work page 2024

  2. [2]

    Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-based clustering based on hierarchical den- sity estimates. In Advances in Knowledge Discovery and Data Mining , pages 160–172, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. 1

    work page 2013

  3. [3]

    Adamv-moe: Adaptive multi-task vision mixture-of- experts

    Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamv-moe: Adaptive multi-task vision mixture-of- experts. In IEEE/CVF International Conference on Com- puter Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 17300–17311. IEEE, 2023. 1, 2

    work page 2023

  4. [4]

    Daxberger, Floris Weers, Bowen Zhang, Tom Gunter, Ruoming Pang, Marcin Eichner, Michael Emmers- berger, Yinfei Yang, Alexander Toshev, and Xianzhi Du

    Erik A. Daxberger, Floris Weers, Bowen Zhang, Tom Gunter, Ruoming Pang, Marcin Eichner, Michael Emmers- berger, Yinfei Yang, Alexander Toshev, and Xianzhi Du. Mobile v-moes: Scaling down vision transformers via sparse mixture-of-experts. CoRR, abs/2309.04354, 2023. 1, 2

    arXiv 2023

  5. [5]

    Editing factual knowledge in language models

    Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pages 6491–6506, Online and Punta Cana, Domini- can Republic, 2021. Association for Computational Linguis- tics. 2

    work page 2021

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 4

    2009

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Rep- resentations, ICLR 20...

    2021

  8. [8]

    Learning factored representations in a deep mixture of ex- perts

    David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of ex- perts. In 2nd International Conference on Learning Rep- resentations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. 2

    work page 2014

Show all 45 references
  1. [9]

    Switch transformers: scaling to trillion parameter models with sim- ple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with sim- ple and efficient sparsity. J. Mach. Learn. Res., 23(1), 2022. 2

    2022

  2. [10]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic, 2021. Associa...

    2021

  3. [11]

    Paca-vit: Learning patch-to- cluster attention in vision transformers

    Ryan Grainger, Thomas Paniagua, Xi Song, Naresh Cuntoor, Mun Wai Lee, and Tianfu Wu. Paca-vit: Learning patch-to- cluster attention in vision transformers. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 ,...

    2023

  4. [12]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts.Neu- ral Comput., 3(1):79–87, 1991. 2

    1991

  5. [13]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 5, 3

    2009

  6. [14]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, I...

    2021

  7. [15]

    M3vit: Mixture-of-experts vision transformer for ef- ficient multi-task learning with model-accelerator co-design

    Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tian- long Chen, Kai Zou, Yu Cheng, Cong Hao, and Zhangyang Wang. M3vit: Mixture-of-experts vision transformer for ef- ficient multi-task learning with model-accelerator co-design. CoRR, abs/2210.14793, 2022. 1, 2

    arXiv

  8. [16]

    Liang, Yiming Cui, Qifan Wang, Tong Geng, Wen- guan Wang, and Dongfang Liu

    James C. Liang, Yiming Cui, Qifan Wang, Tong Geng, Wen- guan Wang, and Dongfang Liu. Clusterformer: clustering as a universal visual learner. In Proceedings of the 37th Inter- national Conference on Neural Information Processing Sys- tems, Red Hook, NY , USA, 2024. Curran Asso...

    2024

  9. [17]

    Expediting large-scale vision transformer for dense predic- tion without fine-tuning

    Weicong Liang, Yuhui Yuan, Henghui Ding, Xiao Luo, Wei- hong Lin, Ding Jia, Zheng Zhang, Chao Zhang, and Han Hu. Expediting large-scale vision transformer for dense predic- tion without fine-tuning. In Advances in Neural Informa- tion Processing Systems 35: Annual Conference o...

    2022

  10. [18]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vi- sion, ICCV 2021, Montreal, QC, Canada, October 10-17, ...

    2021

  11. [19]

    A ConvNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11966–11976, 2022. 5, 3

    2022

  12. [20]

    SGDR: stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.net, 2017. 4

    2017

  13. [21]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 4

    2019

  14. [22]

    Data-free dynamic compression of cnns for tractable effi- ciency

    Lukas Meiner, Jens Mehnert, and Alexandru Condurache. Data-free dynamic compression of cnns for tractable effi- ciency. In Proceedings of the 20th International Joint Con- 9 ference on Computer Vision, Imaging and Computer Graph- ics Theory and Applications - Volume 2: VISAPP ...

    2025

  15. [23]

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision trans- formers see like convolutional neural networks? In Pro- ceedings of the 35th International Conference on Neural In- formation Processing Systems, Red Hook, NY , USA, 202...

    2024

  16. [24]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems 34: Annual Con- ference on Neural Information Processing Systems 20...

    2021

  17. [25]

    Richards, Timo- thy P

    David Raposo, Samuel Ritter, Blake A. Richards, Timo- thy P. Lillicrap, Peter Conway Humphreys, and Adam San- toro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. CoRR, abs/2404.02258,

    arXiv

  18. [26]

    Scaling vision with sparse mix- ture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr ´e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mix- ture of experts. In Proceedings of the 35th International Conference on Neural Information Processing Syst...

    2021

  19. [27]

    Le, Geoffrey E

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Fran...

    2017

  20. [28]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 Ju...

    2021

  21. [29]

    Gomez, Łukasz Kaiser, and Il- lia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Il- lia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 6000–6010, Red Hook, ...

  22. [30]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Thirty-Sixth AAAI Conference on Ar- tificial Intelligence, AAAI 2022, Thirty-Fourth Confere...

    2022

  23. [31]

    Width & depth pruning for vision transform- ers

    Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width & depth pruning for vision transform- ers. In Thirty-Sixth AAAI Conference on Artificial Intelli- gence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, ...

    2022

  24. [32]

    MoEfication: Transformer feed-forward layers are mixtures of experts

    Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. MoEfication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 877–890, Dublin, Ireland, 2022. Association for Com...

    2022

  25. [33]

    Emergent modularity in pre- trained transformers

    Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Emergent modularity in pre- trained transformers. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4066–4083, Toront...

    2023

  26. [34]

    Vision transformer pruning, 2021

    Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning, 2021. 2 10 Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks Supplementary Material /uni0000001c/uni00000018/uni00000008/uni0000001c/uni00000013/uni00000008/uni0000001b/uni00000018/uni000...

    2021

  27. [37]

    Sensitivity to Hyperparameters Since we use a subset of the dataset as samples for expert extraction, to select an appropriate minimum cluster size for different sample sizes, we consider the minimum cluster size relative to the total number of samples as a percent- age. Incre...

  28. [38]

    Sensitivity to Sample Size To evaluate the effect of the sample size on our method, we use the DeiT-S model, applying the same training settings as described in Section 4 of the paper. Table 3 presents the mean Top-1 Accuracy on ImageNet- 1k over three random seeds, along with...

  29. [39]

    [32] rely on weight co-activation graphs and a manually set number of experts

    Comparison to MoEfication Zhang et al. [32] rely on weight co-activation graphs and a manually set number of experts. In contrast, we cluster ac- tivations and let the data determine the number of experts automatically. Their disjoint partitioning contrasts with our variance-b...

  30. [40]

    Our results on ImageNet-1k validate this design, showing performance gains in vision tasks

    routes tokens to the most similar mean weight column, we compute similarities in the input space directly (see Ta- ble 4). Our results on ImageNet-1k validate this design, showing performance gains in vision tasks. 2 Model MACs (G) Parameters (M) Acc. Retention (%) Top-1 Acc. ...

  31. [41]

    Despite structural differences, all variants benefit from fewer MACs, reduced parameter counts and competitive final accuracies

    Generalizability to other Datasets and Ar- chitectures The results in Table 5 and Table 6 confirm that our method generalizes well across both hierarchical and convolution- inspired transformer architectures (Swin-Transformer [18] and ConvNeXt Models [19]), as well as to small...

  32. [42]

    We compare our default method, HDBSCAN, against other density-based methods (DBSCAN, OPTICS) as well as partition-based alternatives (K-Means, BIRCH)

    Effect of Clustering Algorithm To evaluate impact of different clustering algorithms on expert extraction, we conduct experiments on the DeiT-S model using the same training settings as described in Sec- tion 4 and evaluate the resulting model on ImageNet-1k. We compare our de...

  33. [43]

    For the ex- traction strategy, we consider two methods for selecting the hidden neurons of each expert

    Effect of Extraction and Routing Method In order to evaluate the effects of the extraction and routing methods, we perform another ablation study. For the ex- traction strategy, we consider two methods for selecting the hidden neurons of each expert. In the Magnitude-Based ap-...

  34. [44]

    Insights into Routing Distributions We further analyse the routing distribution in Figure 8, which shows the token routing distributions for selected visually similar classes in layer 11, compared to the dis- tribution across all 1,000 ImageNet-1k classes. Specifi- cally, Figu...

  35. [45]

    Notably, the earlier lay- ers do not exhibit any formed clusters, reflecting the more general feature representations in shallower layers

    Insights into Expert Formations Table 9 and 10 show the mean number of experts extracted in each layer for different models. Notably, the earlier lay- ers do not exhibit any formed clusters, reflecting the more general feature representations in shallower layers. Deeper layers...

  36. [2017]

    Curran Associates Inc. 2

  37. [2021]

    OpenReview.net, 2021. 1, 2

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.