CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection

Humaira Ayesha; Junaid Ahmed Sifat; Md Irtiza Hossain

REVIEW 2 major objections 1 minor 15 references

The shared low-rank SVD basis, not the routing mechanism, preserves accuracy when turning frozen Vision Transformers into Mixture-of-Experts models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 01:36 UTC pith:K5HCITCH

load-bearing objection CLEAR-MoE shows a post-training SVD-plus-clustering route to turn frozen ViTs into MoE models while keeping accuracy on the calibration set, with the shared basis doing most of the work. the 2 major comments →

arxiv 2606.28516 v1 pith:K5HCITCH submitted 2026-06-26 cs.CV cs.DC

CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection

Md Irtiza Hossain , Humaira Ayesha , Junaid Ahmed Sifat This is my paper

classification cs.CV cs.DC

keywords Mixture-of-ExpertsVision Transformerpost-training conversionSVD decompositionlayer selectionfrozen modelmodel sparsification

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CLEAR-MoE is a post-training pipeline that converts a frozen Vision Transformer into a sparse Mixture-of-Experts model by scoring FFN layers on sparsity, clusterability, and output sensitivity, then decomposing selected layers into a shared low-rank SVD basis plus per-cluster residual experts via k-means. Lightweight routers are trained on the resulting cluster labels and tokens are dispatched through pluggable CUDA kernels. The central empirical result is that this shared SVD basis accounts for accuracy retention, since random routing, learned routing, and multiple router designs all yield performance within 0.06 percentage points of one another.

Core claim

The shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). This behavior generalizes across five ViT backbones covering 5.7M to 86.6M parameters, with accuracy differences at most 0.10 percentage points from the dense models, and remains stable under changes in SVD rank, expert count, calibration set size, and random seed.

What carries the argument

Shared low-rank SVD basis extracted from selected FFN weight matrices, paired with k-means-derived per-cluster residual experts.

Load-bearing premise

That the three layer-scoring criteria reliably identify FFN layers whose decomposition into a shared low-rank SVD basis plus per-cluster residuals will preserve the original model's behavior on unseen inputs and tasks beyond the calibration set.

What would settle it

If accuracy on a new image-classification dataset drops more than 0.5 percentage points below the dense baseline when the shared-basis model is paired with random routing, the claim that the basis alone preserves behavior would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

The converted model retains 99.9 percent of the dense accuracy on Imagenette with DeiT-Small.
Accuracy stays stable for expert counts from 2 to 8 and across calibration sets of 50 to 500 samples.
The same accuracy retention holds for all tested ViT sizes from 5.7M to 86.6M parameters.
The MoE FFN runs 1.3-1.7 times slower than the dense version on a GTX 960 GPU because dispatch is memory-bound.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The low-rank structure uncovered by the SVD step may be a general property of many transformer FFN layers.
Future speedups would require fused kernels that reduce the memory-bound routing overhead identified in the dispatch microbenchmark.
The finding that routing choice is secondary could motivate simpler expert-selection schemes in other post-training conversion pipelines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

CLEAR-MoE shows a post-training SVD-plus-clustering route to turn frozen ViTs into MoE models while keeping accuracy on the calibration set, with the shared basis doing most of the work.

read the letter

The main point is a four-phase pipeline that scores FFN layers in a frozen ViT by sparsity, clusterability and output sensitivity, decomposes the chosen layers into a shared low-rank SVD basis plus per-cluster residuals, trains lightweight routers on the cluster labels, and runs the result through custom dispatch. On Imagenette it keeps accuracy within 0.03 points of the dense model across five backbones from 5.7 M to 86 M parameters, and the ablations show that changing the router or even using random routing shifts accuracy by at most 0.06 points.

What stands out is the concrete empirical claim that the shared SVD basis, not the router, is what preserves behavior, plus the fact that the same pattern holds across model sizes and hyper-parameter choices. The numbers are reported with small standard deviations and the stability across seeds and calibration sizes is shown directly.

The soft spot is that every measurement, including the layer scoring, the k-means clustering, and all the ablation tables, comes from the same Imagenette calibration distribution. No out-of-distribution images or different tasks are tested, so it is still open whether the decomposition approximates the original mapping on shifted data. The inference overhead from routing is also measured only on one GPU and is reported as 1.3-1.7x slower.

This is aimed at people who want a drop-in way to sparsify existing vision checkpoints for inference. It has enough new empirical content and a clear, reproducible pipeline to deserve a serious referee, though any review will need to press on generalization beyond the calibration set.

Referee Report

2 major / 1 minor

Summary. The manuscript presents CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer into a sparse MoE model without updating backbone weights. FFN layers are scored by sparsity, clusterability, and output sensitivity; selected layers are decomposed into a shared low-rank SVD basis plus per-cluster residual experts via k-means; lightweight routers are trained on cluster labels; and tokens are dispatched through pluggable CUDA kernels. On Imagenette the method retains 99.9% of dense accuracy for DeiT-Small (86.70% vs 86.73%) and comparable retention across five backbones (5.7M–86.6M parameters), with ablations showing that accuracy varies by at most 0.06 pp across random, learned, and alternative routers and remains stable across SVD ranks, expert counts, and calibration sizes.

Significance. If the central empirical finding holds, the work supplies a practical route to sparse ViT inference that avoids any backbone fine-tuning and isolates the shared low-rank SVD basis as the dominant accuracy-preserving component. The reported routing-insensitivity and hyperparameter stability constitute a concrete, falsifiable observation that could guide future MoE design. The multi-backbone coverage and explicit timing measurements are additional strengths.

major comments (2)

[Abstract] Abstract: The claim that the shared SVD basis is the primary factor responsible for accuracy preservation rests on the observation that random routing, learned routing, and three router architectures produce nearly identical results (variation ≤ 0.06 pp). All such ablations, however, are performed exclusively on the Imagenette calibration distribution; without OOD or cross-task results it remains untested whether the decomposition (and the three scoring criteria used to select layers) generalizes beyond calibration statistics.
[Abstract] Abstract: The three layer-scoring criteria (sparsity, clusterability, output sensitivity) and the k-means clustering step are both computed on the calibration set, yet the manuscript supplies no explicit formulas, pseudocode, or protocol for these quantities. This omission directly affects reproducibility of the layer-selection step that underpins the entire pipeline and the claim that the resulting experts preserve original behavior on unseen inputs.

minor comments (1)

[Abstract] Abstract: The reported standard deviation (±0.02%) implies repeated runs, but the number of seeds, exact statistical test, and whether the same calibration set was reused across runs are not stated.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address the two major comments point by point below. We agree that explicit formulas and pseudocode are required for reproducibility and will add them. On the generalization point, we maintain that the current experiments are scoped to the calibration distribution and multi-backbone evaluation, which is consistent with the manuscript's claims.

read point-by-point responses

Referee: [Abstract] Abstract: The claim that the shared SVD basis is the primary factor responsible for accuracy preservation rests on the observation that random routing, learned routing, and three router architectures produce nearly identical results (variation ≤ 0.06 pp). All such ablations, however, are performed exclusively on the Imagenette calibration distribution; without OOD or cross-task results it remains untested whether the decomposition (and the three scoring criteria used to select layers) generalizes beyond calibration statistics.

Authors: The ablations on routing variants were intentionally performed on the Imagenette calibration set to isolate the contribution of the shared SVD basis versus routing. The manuscript's central empirical claim is that, within this distribution, the shared basis dominates accuracy preservation, as evidenced by the ≤0.06 pp variation. We do not claim or test generalization to OOD or cross-task settings in the current work; the multi-backbone results (DeiT-Tiny through ViT-Base) demonstrate consistency of the pipeline across model scales but remain within the same evaluation protocol. We therefore do not plan to expand the scope with new OOD experiments for this revision. revision: no
Referee: [Abstract] Abstract: The three layer-scoring criteria (sparsity, clusterability, output sensitivity) and the k-means clustering step are both computed on the calibration set, yet the manuscript supplies no explicit formulas, pseudocode, or protocol for these quantities. This omission directly affects reproducibility of the layer-selection step that underpins the entire pipeline and the claim that the resulting experts preserve original behavior on unseen inputs.

Authors: We agree that the absence of explicit definitions and pseudocode for the three scoring criteria and the clustering procedure limits reproducibility. In the revised manuscript we will add: (i) the precise mathematical definitions of sparsity (fraction of near-zero activations), clusterability (silhouette score on activation vectors), and output sensitivity (L2 norm of output change under perturbation); (ii) the exact protocol for computing these scores on the calibration set; and (iii) pseudocode for the layer-selection and k-means decomposition steps. These additions will be placed in the Methods section and an appendix. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical pipeline with direct held-out measurements.

full rationale

The manuscript contains no mathematical derivations, uniqueness theorems, or first-principles predictions. It presents a four-phase empirical pipeline (layer scoring, SVD decomposition + k-means, router training, dispatch) and reports accuracy numbers plus ablations measured on held-out Imagenette data. The key observation that random routing matches learned routing is a direct empirical comparison on the same test split, not a fitted parameter renamed as a prediction. No self-citation chains, ansatzes smuggled via prior work, or self-definitional reductions appear. The work is therefore self-contained against its external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The approach is driven by empirical calibration and clustering rather than explicit axioms or fitted parameters; SVD rank and expert count are varied experimentally but not optimized against a loss that would create circularity.

pith-pipeline@v0.9.1-grok · 5898 in / 1330 out tokens · 54983 ms · 2026-06-30T01:36:49.206432+00:00 · methodology

0 comments

read the original abstract

We present CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights. The pipeline (i) scores feed-forward network (FFN) layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis and per-cluster residual experts using k-means clustering; (iii) trains lightweight routers supervised by cluster labels; and (iv) dispatches tokens through pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of the dense model's accuracy (86.70 +/- 0.02% versus 86.73%). Extensive ablation studies reveal a consistent empirical finding: the shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). Accuracy also remains stable across different SVD ranks, expert counts (2-8), calibration set sizes (50-500), and random seeds. This behavior generalizes across five ViT backbones (DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Small, and ViT-Base), covering models from 5.7M to 86.6M parameters, with accuracy differences <= 0.10 percentage points from their dense counterparts. On a GTX 960 GPU, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense implementation. A dispatch microbenchmark further shows that routing is an order of magnitude more memory-bound than expert matrix multiplications, identifying fused dispatch kernels as a promising direction for future optimization.

Figures

Figures reproduced from arXiv: 2606.28516 by Humaira Ayesha, Junaid Ahmed Sifat, Md Irtiza Hossain.

**Figure 2.** Figure 2: Roofline for GTX 960. Blue = compute-bound; orange/red = memory [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Mean per-token routing entropy (nats) per expertized layer. Dashed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: Accuracy gap (∆pp relative to dense) vs. router training across 13 configurations. D5 (random, no trained router) at x=0; D6 and cross-backbone routers at measured routing accuracy. No strong monotonic relationship is visible, consistent with the hypothesis that shared-basis quality governs accuracy regardless of routing precision. Calibration size (N ∈ {50, 100, 200, 500}, multiple subsets for N<500): ac… view at source ↗

**Figure 5.** Figure 5: Accuracy ∆ vs. dense for five ViT backbones under D5 (random) and D6 (learned linear) routing (Imagenette, E=4, N=200, seed 42). All deltas within |∆|≤0.10 pp. D6 provides marginal numerical advantage in 4 of 5 backbones; D5 outperforms D6 only on ViT-B (highest load skew, 0.150). (2 TB/s) or H100 (3.35 TB/s) class hardware are the plausible candidates. Practical design guidelines. When to use CLEAR-MoE. C… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages · 1 internal anchor

[1]

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,”CoRR, vol. abs/2010.11929, 2020. [Online]. Available: https://arxiv.org/abs/2010.11929

work page internal anchor Pith review Pith/arXiv arXiv 2010
[2]

Scaling vision with sparse mixture of experts,

C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” 06 2021

2021
[3]

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

G. Hinton, N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 01 2017

2017
[4]

Adamv-moe: Adaptive multi-task vision mixture-of-experts,

T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y . Li, “Adamv-moe: Adaptive multi-task vision mixture-of-experts,” 10 2023, pp. 17 300–17 311

2023
[5]

Moefication: Transformer feed-forward layers are mixtures of experts,

Z. Zhang, Y . Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” 01 2022, pp. 877–890

2022
[6]

Exploiting activation sparsity with dense to dynamic-k mixture-of- experts conversion,

F. Szatkowski, B. W’ojcik, M. Pi’orczy’nski, and S. Scardapane, “Exploiting activation sparsity with dense to dynamic-k mixture-of- experts conversion,”Advances in Neural Information Processing Systems 37, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID: 263831552

2023
[7]

Efficient data driven mixture-of-expert extraction from trained networks,

U. Berisha, J. Mehnert, and A. P. Condurache, “Efficient data driven mixture-of-expert extraction from trained networks,” 2025. [Online]. Available: https://arxiv.org/abs/2505.15414

work page arXiv 2025
[8]

Sparse upcycling: Training mixture-of-experts from dense checkpoints,

A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. Ruiz, B. Mustafa, J. Ainslie, Y . Tay, M. dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” 12 2022

2022
[9]

Dynamicvit: Efficient vision transformers with dynamic token sparsification,

Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynamicvit: Efficient vision transformers with dynamic token sparsification,” 06 2021

2021
[10]

A-vit: Adaptive tokens for efficient vision transformer,

H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” 06 2022, pp. 10 799–10 808

2022
[11]

Tutel: Adaptive mixture-of-experts at scale,

C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y . Xiong, “Tutel: Adaptive mixture-of-experts at scale,” 06 2022

2022
[12]

Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,

W. Cui, S. Jiao, T. University, Z. Han, M. Research, Y . Wang, N. Zheng, L. Ma, Y . Yang, F. Yang, J. Xue, L. Qiu, L. Zhou, A. Quan, H. Tan, M. Guo, L. Ouyang, and Q. Chen, “Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,” 07 2023

2023
[13]

arXiv preprint arXiv:2404.19429 (2024)

C. Jiang, Y . Tian, Z. Jia, S. Zheng, C. Wu, and Y . Wang, “Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,”ArXiv, vol. abs/2404.19429,

work page arXiv
[14]

Available: https://api.semanticscholar.org/CorpusID: 269456940

[Online]. Available: https://api.semanticscholar.org/CorpusID: 269456940
[15]

Mixture-of-experts with expert choice routing,

Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” 02 2022. APPENDIXA METHODOLOGYDETAILS A. Glossary of Key Terms Table A1 defines domain-specific terminology used through- out the paper. TABLE A1 DOMAIN-SPECIFIC TERMINOLOGY. Term Definition Token Image patch embedd...

2022

[1] [1]

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,”CoRR, vol. abs/2010.11929, 2020. [Online]. Available: https://arxiv.org/abs/2010.11929

work page internal anchor Pith review Pith/arXiv arXiv 2010

[2] [2]

Scaling vision with sparse mixture of experts,

C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” 06 2021

2021

[3] [3]

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

G. Hinton, N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 01 2017

2017

[4] [4]

Adamv-moe: Adaptive multi-task vision mixture-of-experts,

T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y . Li, “Adamv-moe: Adaptive multi-task vision mixture-of-experts,” 10 2023, pp. 17 300–17 311

2023

[5] [5]

Moefication: Transformer feed-forward layers are mixtures of experts,

Z. Zhang, Y . Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” 01 2022, pp. 877–890

2022

[6] [6]

Exploiting activation sparsity with dense to dynamic-k mixture-of- experts conversion,

F. Szatkowski, B. W’ojcik, M. Pi’orczy’nski, and S. Scardapane, “Exploiting activation sparsity with dense to dynamic-k mixture-of- experts conversion,”Advances in Neural Information Processing Systems 37, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID: 263831552

2023

[7] [7]

Efficient data driven mixture-of-expert extraction from trained networks,

U. Berisha, J. Mehnert, and A. P. Condurache, “Efficient data driven mixture-of-expert extraction from trained networks,” 2025. [Online]. Available: https://arxiv.org/abs/2505.15414

work page arXiv 2025

[8] [8]

Sparse upcycling: Training mixture-of-experts from dense checkpoints,

A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. Ruiz, B. Mustafa, J. Ainslie, Y . Tay, M. dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” 12 2022

2022

[9] [9]

Dynamicvit: Efficient vision transformers with dynamic token sparsification,

Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynamicvit: Efficient vision transformers with dynamic token sparsification,” 06 2021

2021

[10] [10]

A-vit: Adaptive tokens for efficient vision transformer,

H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” 06 2022, pp. 10 799–10 808

2022

[11] [11]

Tutel: Adaptive mixture-of-experts at scale,

C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y . Xiong, “Tutel: Adaptive mixture-of-experts at scale,” 06 2022

2022

[12] [12]

Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,

W. Cui, S. Jiao, T. University, Z. Han, M. Research, Y . Wang, N. Zheng, L. Ma, Y . Yang, F. Yang, J. Xue, L. Qiu, L. Zhou, A. Quan, H. Tan, M. Guo, L. Ouyang, and Q. Chen, “Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,” 07 2023

2023

[13] [13]

arXiv preprint arXiv:2404.19429 (2024)

C. Jiang, Y . Tian, Z. Jia, S. Zheng, C. Wu, and Y . Wang, “Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,”ArXiv, vol. abs/2404.19429,

work page arXiv

[14] [14]

Available: https://api.semanticscholar.org/CorpusID: 269456940

[Online]. Available: https://api.semanticscholar.org/CorpusID: 269456940

[15] [15]

Mixture-of-experts with expert choice routing,

Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” 02 2022. APPENDIXA METHODOLOGYDETAILS A. Glossary of Key Terms Table A1 defines domain-specific terminology used through- out the paper. TABLE A1 DOMAIN-SPECIFIC TERMINOLOGY. Term Definition Token Image patch embedd...

2022