Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sub-MoE claims that merging experts in a shared SVD subspace preserves specialized knowledge, keeping 94–96% of Mixtral-8x7B accuracy after dropping 25% of experts.

desk verdict Broad, plausible empirical recipe for MoE compression, but the central subspace-merge mechanism is either algebraically identical to weighted averaging or rests on an unstated truncation rank. read the letter →

arxiv 2506.23266 v1 pith:H7FTRV4K submitted 2025-06-29 cs.LG

Lujun Li , Zhu Qiyuan , Jiacheng Wang , Wei Li , Hao Gu , Sirui Han , Yike Guo This is my paper
classification cs.LG
keywords mixtureofexpertsexpertmergingsingularvaluedecompositionsubspacealignmentLLMcompressionfrequency-awareK-meansclusteringparameterconflict
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sub-MoE is a compression method for mixture-of-experts language models that replaces several expert networks with one merged expert, without fine-tuning. The paper's central claim is that expert weights should not be averaged directly; instead, experts in the same functional cluster are first projected into a common subspace via a joint singular value decomposition, and only the expert-specific right-singular components are frequency-weighted and merged. This is said to reduce parameter conflicts caused by expert specialization and to preserve more specialized knowledge than standard expert merging or pruning. On Mixtral-8x7B, the method reports keeping 94–96% of zero-shot accuracy with 25% of experts removed and 86–87% with 50% removed, with similar patterns on Qwen and DeepSeek MoE models. A calibration set of 128 WikiText-2 samples is needed for clustering and merging.

What carries the argument

The central mechanism is Experts Union Decomposition: the joint SVD of the concatenated expert weight matrices in one cluster, yielding a shared left-singular matrix $U$, a singular-value matrix $\Sigma$, and per-expert right-singular matrices $V^{(i)}$. The method proceeds in three steps: K-means clustering of experts by cosine similarity of their outputs, frequency-based merging of the $V^{(i)}$ matrices using router activation counts, and reconstruction of one weight matrix per cluster from $U$, $\Sigma$, and the merged $V$. The shared $U$ is the component claimed to eliminate conflicting parameters, while the frequency weights are the component claimed to preserve the most-used experts.

What would settle it

Run Sub-MoE on a cluster with a full-rank joint SVD, meaning no singular values are truncated, and compare the reconstructed merged weight matrix to the direct frequency-weighted average of the original expert matrices: if the two matrices are equal, the claimed subspace mechanism has no effect beyond clustering and frequency weighting. Separately, re-run clustering and merging with a held-out calibration set from a different domain and check whether the reported 94–96% and 86–87% retention figures on Mixtral persist.

Watch

Extended reading notes

Core claim

The paper's discovery is that the obstacle to expert merging—low inter-expert similarity, which makes direct weight averaging destructive—can be addressed by decomposing the concatenated weights of a cluster of experts as $U\Sigma[V^{(1)};\dots;V^{(n)}]^T$, sharing the left basis $U$ across all experts in the cluster, and merging only the $V$ blocks. The merged expert is reconstructed as $W_{\text{merged}} = U\Sigma[V_{\text{merged}}]^T$. The paper argues that this forces the experts to be expressed in one common coordinate system before fusion, so the merged parameters inherit a shared functional subspace rather than averaging away divergent representations. It reports consistent gains over frequency pruning, output pruning, MC-SMoE, and HC-SMoE across Mixtral, Qwen1.5-MoE, Qwen3-MoE, and DeepSeek-MoE, and it presents an extended version, Sub-MoE$^\dagger$, that adds activation-aware truncation for intra-expert compression.

Load-bearing premise

The load-bearing premise is that sharing $U$ and merging only $V$ is genuinely different from merging the original weights; if the joint SVD is run at full rank with no singular values dropped, the reconstruction is algebraically equivalent to a frequency-weighted average of the original experts, so the subspace would do no conflict-reduction work. The method also depends on 128 WikiText-2 samples being representative of the evaluation tasks.

Editorial extensions

If this is right

  • If the claim holds, MoE LLMs can be compressed by removing a quarter to half of their experts without fine-tuning, keeping roughly 94–96% and 86–87% of zero-shot accuracy on Mixtral-8x7B.
  • Sub-MoE$^\dagger$ extends the same subspace merging to intra-expert compression, reporting 1.1–1.3$\times$ throughput on 8x H800 GPUs and more stable performance than intra-expert truncation applied after standard merging.
  • Because the method needs only a small calibration set and no training, it can be applied to models whose training pipeline is unavailable.
  • Frequency-based $V$-merging gives more weight to experts activated often, so the compressed model should preserve behavior on common inputs more than on rare ones.
  • The method reports consistent gains across four MoE architectures, suggesting the subspace-alignment mechanism is not tied to one router design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: at full rank, the joint SVD reconstruction is algebraically the frequency-weighted average of the original experts, so the reported gains may be driven by rank truncation and clustering rather than by the shared $U$ per se; an ablation that fixes the same clusters and compares direct weighted averaging with the Sub-MoE reconstruction at the same truncation would isolate the me
  • Editorial inference: the method's dependence on 128 WikiText-2 samples for clustering and frequencies suggests a stress test the paper does not report—vary the calibration distribution and measure retention.
  • Editorial inference: the same merge-in-a-shared-subspace recipe might transfer to other parameter-sharing settings, such as merging layers or attention heads, where the objects being merged have low similarity; this is a natural extension the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Sub-MoE, a training-free MoE compression method with two stages: adaptive K-means clustering of experts based on cosine similarity of their outputs, and 'subspace expert merging' that applies a joint SVD to concatenated expert weights, merges the per-expert V components with frequency weights, and reconstructs a single merged expert from shared U, Sigma, and the merged V. The method is evaluated on Mixtral-8x7B, Qwen1.5-MoE-A2.7B, Qwen3-30B-A3B, and DeepSeek-MoE-16B, against expert pruning and merging baselines, and is extended to Sub-MoE-dagger with activation-aware truncated SVD for intra-expert compression. The paper reports substantial gains, such as maintaining 0.58 average accuracy with 4 experts on Mixtral versus 0.51 for HC-SMoE, and includes ablations over clustering settings, merging settings, and calibration size.

Significance. If the subspace-alignment mechanism were genuine and fully specified, Sub-MoE would be a practically valuable training-free MoE compression method, and the broad empirical coverage across four models and multiple baselines is a real strength. The paper also provides useful ablations and an analysis of calibration sample size. However, the central algebraic claim is currently under-specified: at full rank the proposed subspace merge reduces exactly to frequency-weighted direct averaging, so the claimed conflict-reduction role of the shared U is not established. With a clarified truncation rank and an ablation that isolates the subspace effect, the empirical results could support a useful contribution; as written, the mechanism is either vacuous or unverified.

major comments (4)
  1. [Section 3.3, Eqs. (5)-(8)] The manuscript never states the rank r or a truncation step for the base Sub-MoE method. If the SVD in Eq. (5) is exact (full rank), then, under the intended horizontal concatenation, each expert satisfies W_i = U Sigma V_i^T, and substituting Eq. (7) into Eq. (8) gives W_merged = U Sigma (sum_i f_i V_i^T)/(sum_i f_i) = (sum_i f_i W_i)/(sum_i f_i), which is exactly the frequency-weighted direct averaging of Eq. (4). In that case the shared U performs no conflict reduction; it is merely a basis change. Section 3.3 describes no truncation, and the implementation note that singular values are multiplied into U is consistent with full rank. Thus the paper's central mechanism is either algebraically equivalent to vanilla weighted averaging or, if a truncated SVD is actually used, empirically unverified because the truncation rank is not reported. Please specify the rank and truncation used in every reported experiment, and add an ablation comparing full-rank subspace merging against direct frequency-weighted averaging.
  2. [Eq. (5), Section 3.3] The concatenation geometry is inconsistent and load-bearing. The text says 'concatenate their weight matrices vertically', but the notation U in R^{O x r}, V in R^{r x nI}, with V partitioned into n blocks, only works for horizontal concatenation [W^(1), ..., W^(n)]. Under vertical concatenation, the SVD of the nO x I matrix has a single common V^T, and one would need to partition U into row blocks U_i to reconstruct W_i = U_i Sigma V^T, so the per-expert V_i used in Eq. (7) is not well-defined. Please clarify the concatenation direction and the corresponding block structure; this is essential for reproducing the method and for the correctness of Eqs. (7)-(8).
  3. [Table 3(B), Section 4.4] The ablation does not isolate the proposed subspace-sharing mechanism. The 'Vanilla SVD' row is not defined in the experimental setup, and no baseline of direct frequency-weighted averaging is reported. Without a comparison between full-rank Sub-MoE and direct averaging, and without truncated-rank variants of the joint SVD, the reported gains over HC-SMoE cannot be attributed to subspace alignment; they could come entirely from the expert clustering and frequency-weighting recipe. Please define 'Vanilla SVD', add a direct frequency-weighted average baseline, and vary the SVD truncation rank to show that the shared-subspace operation itself contributes to the results.
  4. [Section 3.2, Multi-layer Adaptive Allocation] The adaptive allocation of cluster counts is underspecified. The paper states that experts on multiple MoE layers are jointly clustered 'while maintaining a target overall compression ratio' and that the process 'automatically' determines the number of clusters per layer, but no algorithm, objective, or mapping from the target compression ratio to the layer-wise cluster count k_l is given. Table 3(A) reports 1-, 2-, and 3-layer variants with different outcomes, yet the mechanism for choosing layer-wise cluster counts is not described, which prevents reproduction of the method's core compression schedule.
minor comments (6)
  1. [Abstract and Introduction] The reported retained accuracy for Mixtral-8x7B is inconsistent: the abstract says 96% and 86% for 25% and 50% expert reduction, while the introduction says 94% and 87%, and Table 1 implies 0.64/0.67 = 95.5% and 0.58/0.67 = 86.6%. Please align these numbers.
  2. [Section 4.1 and Table 1] The model list in Section 4.1 mentions Qwen3-235B-A22B, but Table 1 reports results for Qwen3-30B-A3B. Please clarify which Qwen3 model was actually evaluated and correct the setup description.
  3. [Table 2 caption] The caption says 'Performance of Sub-MoE and MC-SMoE', but the proposed method rows are labeled Sub-MoE-dagger; please adjust the caption to match the table entries.
  4. [Section 1, paragraph 2] There is a typo in 'expert-specific V T -matrixs'; it should read 'V^T matrices'.
  5. [Section 3.3, paragraph after Eq. (7)] The sentence 'while still preserving capabilities from experts' is incomplete; it should say something like 'while still preserving the specialized capabilities of less frequent experts'.
  6. [Reference [45]] Reference [45] is cited for Qwen3-MoE but points to the Qwen2.5 technical report; please verify and update the citation.

Circularity Check

1 steps flagged · score 8.0 of 10

Subspace merging is algebraically identical to frequency-weighted direct averaging unless the SVD is truncated; the paper never specifies truncation for the base method, so the central conflict-reduction claim reduces to Eq. (4) by construction.

  1. renaming known result [Section 3.3, Equations (4)–(8) (Subspace Expert Merging)]
    "SVD([W(1); W(2); . . .; W(n)]) = U Σ[V(1); V(2); . . .; V(n)]^T ... Vmerged = Σ_{i∈Q} f(Vi)·Vi / Σ_{i∈Q} f(Vi) ... The final merged expert weights are constructed as: Wmerged = U Σ[Vmerged]^T, alongside Eq. (4): Wmerged = Σ_{i=1}^n αi W(i)."

    If the joint SVD in Eq. (5) is exact (no rank truncation), each original expert satisfies W_i = UΣV_i^T. Substituting Eq. (7) into Eq. (8) then gives W_merged = UΣ(Σ_i f_i V_i^T)/(Σ_i f_i) = (Σ_i f_i W_i)/(Σ_i f_i), which is precisely the frequency-weighted version of the 'vanilla' direct average in Eq. (4). The shared U cancels identically, so the claimed subspace alignment performs no conflict reduction; the method is a basis-change restatement of direct frequency-weighted averaging. The paper never states a truncation rank for the base Sub-MoE, and truncation is introduced only for Sub-MoE† in Section 3.4 (Eqs. 10–12). Thus the central novelty reduces by construction unless an unstated rank truncation is assumed.

full rationale

The paper's central mechanism is Eq. (5)–(8): joint SVD of the expert group, frequency-weighted averaging of the V blocks, and reconstruction with one shared U. Because no truncation rank is specified for the base method, the equations are only well-defined at full rank, where the algebra forces W_merged to equal the frequency-weighted direct average of the original expert weights. The claimed 'shared subspace' is then purely notational, not an independent mechanism. The paper does contain external benchmark comparisons and ablations, but the ablations do not isolate full-rank versus truncated union SVD against a direct-average control, so they cannot salvage the claimed mechanism. I did not find other load-bearing circular steps: the self-citations (e.g., D2-MoE) are background context, not uniqueness arguments, and the experimental comparisons are against external baselines. The central derivation, however, is vacuous as written: the subspace merging step is equivalent by construction to the vanilla averaging it claims to supersede.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by data-derived statistics (frequency, similarity, calibration set) and by the unstated rank assumption. There are no invented physical entities. The core mathematical risk is that the subspace merge is a reparameterization of direct weighted averaging unless truncation is explicitly imposed, which the paper never does for the base method.

free parameters (4)
  • per-expert activation frequency f(V_i) = measured on 128-sample WikiText-2 calibration set (Eq 6)
    Serves as the merging weight in Eq (7). It is a data-derived statistic rather than an independently derived constant, and its transferability to other domains is assumed.
  • layer-wise cluster count k_l = determined by target overall compression ratio through adaptive allocation
    The number of merged experts per layer is a hand-chosen target (25 or 50 percent reduction) that directly sets K-means k; the paper offers no automatic selection rule.
  • calibration dataset size = 128 samples of 2048 tokens
    Chosen by hand; Figure 4 shows perplexity improves as sample count increases to 128 but not beyond, indicating sensitivity to this choice.
  • Sub-MoE dagger intra-expert truncation ratio = 10, 20, and 30 percent in Table 2
    Free compression ratio chosen per deployment scenario; no principled selection criterion is provided.
assumptions (5)
  • standard math Singular value decomposition of the concatenated expert matrix exactly reconstructs each expert as U Sigma V_i^T (Eq 5 and Eq 8).
    Invoked throughout Section 3.3; requires consistent matrix dimensions, which the paper does not specify clearly (vertical versus horizontal concatenation).
  • domain assumption Expert output similarity measured on the calibration set reflects functional similarity on the evaluation distribution.
    Section 3.2 uses cosine similarity on 128 WikiText-2 samples; the limitations section admits dependence on calibration data.
  • domain assumption Router activation frequencies measured on the calibration set are representative for weighting merged experts.
    Eq (6) estimates f(V_i) on calibration tokens; no guarantee is given for downstream tasks or other domains.
  • ad hoc to paper Merging experts in a shared subspace with a single merged V produces a valid expert without gradient updates.
    This is the core heuristic claim of the method; the paper provides no proof of the optimization in Eq (9).
  • standard math If rank r is full, U Sigma V_merged^T equals the frequency-weighted arithmetic average of original experts, so any benefit must come from clustering and weights, not subspace alignment.
    Derivable from Eqs (5) through (8); this is why the rank and truncation assumption is load-bearing for the claimed mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging." pith.science (2026). https://pith.science/paper/H7FTRV4K

@misc{pith2026250623266,
  author       = {Pith},
  title        = {Pith review of: Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7FTRV4K}},
  note         = {Machine review of arXiv:2506.23266}
}
abstract

Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges. Although recent expert merging methods promise greater efficiency by consolidating multiple experts, they are fundamentally hindered by parameter conflicts arising from expert specialization. In this paper, we present Sub-MoE, a novel MoE compression framework via Subspace Expert Merging. Our key insight is to perform joint Singular Value Decomposition (SVD) on concatenated expert weights, reducing conflicting parameters by extracting shared $U$-matrices while enabling effective merging of the expert-specific $V$ components. Specifically, Sub-MoE consists of two innovative phases: (1) Adaptive Expert Clustering, which groups functionally coherent experts via K-means clustering based on cosine similarity of expert outputs; and (2) Subspace Expert Merging, which first enforces Experts Union Decomposition to derive the shared $U$-matrix across experts in the same group, then pursues frequency-based merging for individual $V$-matrices, and finalizes expert reconstruction using the merged $V$-matrix. In this way, we align and fuse experts in a shared subspace, and can be extended with intra-expert compression for further inference optimization. Extensive experiments on Mixtral, DeepSeek, and Qwen-1.5|3 MoE LLMs demonstrate that our Sub-MoE significantly outperforms existing expert pruning and merging methods. Notably, our Sub-MoE maintains 96\%|86\% of original performance with 25\%|50\% expert reduction on Mixtral-8x7B in zero-shot benchmarks. Code will be released at https://github.com/lliai/MoERazor.

Figures

Figures reproduced from arXiv: 2506.23266 by the authors.

Figure 1
Figure 1. Overview of our Sub-MoE framework. The process consists of two main stages: (1) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Cosine similarity of output of original expert (left) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Left: Comparison of Sub-MoE† against SVD-LLM [41], NAEE [28], MoE-I2 [46], MoE￾SVD [24], D2 -MoE [11] on 20% compressed Mixtral 6x7B. Middle: Effect of calibration sample size on perplexity. Right: Trade-off between expert count, memory usage, and accuracy for Mixtral MoE. approach provides sufficient flexibility for identifying functional relationships while maintaining manageable cluster sizes. In contrast, 1-laye… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    MoE experts in pretrained Transformers exhibit functional decorrelation with near-zero Jacobian alignment yet occupy partially overlapping representation subspaces, with routing sparsity modulating the geometry.

  2. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    cs.LG 2026-03 conditional novelty 7.0 of 10

    EvoESAP uses evolutionary search guided by a speculative-decoding-inspired ESAP metric to discover non-uniform layer-wise sparsity allocations for MoE expert pruning, improving generation accuracy up to 19.6% at 50% sparsity.

  3. PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

    cs.LG 2025-11 conditional novelty 7.0 of 10

    A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.

  4. Pruning and Distilling Mixture-of-Experts into Dense Language Models

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    A systematic MoE-to-dense conversion via expert scoring, grouping, and distillation yields +6.3 pp average accuracy over dense-to-dense pruning at matched parameter count on tested models.

  5. Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Dynamic expert clustering plus shared-base, low-rank residual compression and two-stage routing cuts MoE parameter count by about 80% with a modest quality cost on GLUE and WikiText-103.

  6. Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

    cs.LG 2024-08 accept novelty 4.0 of 10

    The paper introduces a new taxonomy for model merging methods and reviews their applications in LLMs, MLLMs, continual learning, multi-task learning, and other subfields while outlining open challenges.

Reference graph

Works this paper leans on

48 extracted references · 33 canonical work pages · cited by 6 Pith papers

  1. [1]

    The fifth pascal recognizing textual entailment challenge

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009. 7

    work page 2009

  2. [2]

    Shortcut-connected expert parallelism for accelerating mixture-of-experts

    Weilin Cai, Juyong Jiang, Le Qin, Junwei Cui, Sunghun Kim, and Jiayi Huang. Shortcut-connected expert parallelism for accelerating mixture-of-experts. arXiv preprint arXiv:2404.05019, 2024. 3

    arXiv 2024

  3. [3]

    Retraining-free merging of sparse mixture-of-experts via hierarchical clustering

    I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al. Retraining-free merging of sparse mixture-of-experts via hierarchical clustering. arXiv preprint arXiv:2410.08589, 2024. 2, 3, 7

    arXiv 2024

  4. [4]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers)...

    work page 2019

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. 7

    arXiv 2018

  6. [6]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024. 3, 7

    arXiv 2024

  7. [7]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. 1

    work page 2024

  8. [8]

    Pruner-zero: Evolving symbolic pruning metric from scratch for large language models

    Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In ICML, 2024. 1

    work page 2024

Show all 48 references
  1. [9]

    Stbllm: Breaking the 1-bit barrier with structured binary llms

    Peijie Dong, Lujun Li, Yuedong Zhong, Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Yike Guo, et al. Stbllm: Breaking the 1-bit barrier with structured binary llms. In ICLR, 2025. 3

    2025

  2. [10]

    A framework for few-shot language model evaluation, 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, et al. A framework for few-shot language model evaluation, 2024. 7

    2024

  3. [11]

    Delta decompression for moe-based LLMs compression

    Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based LLMs compression. In F orty-second International Conference on Machine Learning, 2025. 2, 3, 6, 8, 9

    2025

  4. [12]

    Demystifying the compression of mixture-of- experts through a unified framework

    Shwai He, Daize Dong, Liang Ding, and Ang Li. Demystifying the compression of mixture-of- experts through a unified framework. arXiv preprint arXiv:2406.02500, 2024. 3, 7

    2024 arXiv

  5. [13]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. 7

    2021

  6. [14]

    Manifold learning for parameter reduction

    Alexander Holiday, Mahdi Kooshkbaghi, Juan M Bello-Rivas, C William Gear, Antonios Zagaris, and Ioannis G Kevrekidis. Manifold learning for parameter reduction. Journal of computational physics, 392:419–431, 2019. 6 10

    2019

  7. [15]

    Mixture compressor for mixture-of-experts LLMs gains more

    Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QI. Mixture compressor for mixture-of-experts LLMs gains more. In The Thirteenth International Conference on Learning Representations , 2025. 3

    2025

  8. [16]

    Ikotun, Absalom E

    Abiodun M. Ikotun, Absalom E. Ezugwu, Laith Abualigah, Belal Abuhaija, and Jia Heming. K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data. Information Sciences, 622:178–210, 2023. 4

    2023

  9. [17]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. UAI, 2018. 3

    2018

  10. [18]

    Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, et al. Mixtral of experts, 2024. 3, 7

    2024

  11. [19]

    Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping

    Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, and Yida Wang. Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping. arXiv preprint arXiv:2404.19429, 2024. 1

    2024 arXiv

  12. [20]

    Stun: Structured- then-unstructured pruning for scalable moe pruning

    Jaeseong Lee, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He, et al. Stun: Structured- then-unstructured pruning for scalable moe pruning. arXiv preprint arXiv:2409.06211, 2024. 3

    2024 arXiv

  13. [21]

    Discovering sparsity allocation for layer-wise pruning of large language models

    Lujun Li, Peijie, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. In NeuIPS, 2024. 1

    2024

  14. [22]

    Branch-train-merge: Embarrassingly parallel training of expert language models

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. 2022. 3

    2022

  15. [23]

    Merge, then compress: Demystify efficient SMoe with hints from its routing policy

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. 2024. 2, 3, 7

    2024

  16. [24]

    Lee, Shengjie Sun, Wei Xue, and Yike Guo

    Wei Li, Lujun Li, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Structured mixture-of-experts LLMs compression via singular value decomposition. In ICML,

  17. [25]

    Als: Adaptive layer sparsity for large language models via activation correlation assessment

    Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Als: Adaptive layer sparsity for large language models via activation correlation assessment. In NeuIPS, 2024. 1

    2024

  18. [26]

    Pruning via merging: Compressing llms via manifold alignment based layer merging

    Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Xi Chen, Cunhang Fan, et al. Pruning via merging: Compressing llms via manifold alignment based layer merging. arXiv preprint arXiv:2406.16330, 2024. 6

    2024 arXiv

  19. [27]

    Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs

    Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945,

    arXiv

  20. [28]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of- experts large language models

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of- experts large language models. In ACL, 2024. 2, 3, 9

    2024

  21. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. 7

    2018

  22. [30]

    Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts, 2024

    Alexandre Muzio, Alex Sun, and Churan He. Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts, 2024. 2, 3

    2024

  23. [31]

    Libmoe: A library for comprehensive benchmarking mixture of experts in large language models

    Nam V Nguyen, Thong T Doan, Luong Tran, Van Nguyen, and Quang Pham. Libmoe: A library for comprehensive benchmarking mixture of experts in large language models. arXiv preprint arXiv:2411.00918, 2024. 1 11

    2024 arXiv

  24. [32]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. 2019. 7

    2019

  25. [33]

    Revisiting smoe language models by evaluating inefficiencies with task specific expert pruning

    Soumajyoti Sarkar, Leonard Lausen, V olkan Cevher, Sheng Zha, Thomas Brox, and George Karypis. Revisiting smoe language models by evaluating inefficiencies with task specific expert pruning. arXiv preprint arXiv:2409.01483, 2024. 3

    2024 arXiv

  26. [34]

    Se-moe: A scalable and efficient mixture- of-experts distributed training and inference system

    Liang Shen, Zhihua Wu, WeiBao Gong, Hongxiang Hao, Yangfan Bai, HuaChao Wu, Xinxuan Wu, Jiang Bian, Haoyi Xiong, Dianhai Yu, et al. Se-moe: A scalable and efficient mixture- of-experts distributed training and inference system. arXiv preprint arXiv:2205.10034, 2022. 1

    2022 arXiv

  27. [35]

    Model fusion via optimal transport

    Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. NeurIPS, 2020. 6

    2020

  28. [36]

    Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023. 1

    2023

  29. [37]

    Optimizing mode connectivity via neuron alignment

    Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. NeurIPS, 2020. 6

    2020

  30. [38]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024

    Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. 3, 7

    2024

  31. [39]

    A global geometric framework for nonlinear dimensionality reduction

    Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319–2323, 2000. 6

    2000

  32. [40]

    Parameter efficient neural networks with singular value decomposed kernels

    David Vander Mijnsbrugge, Femke Ongenae, and Sofie Van Hoecke. Parameter efficient neural networks with singular value decomposed kernels. IEEE Transactions on Neural Networks and Learning Systems, 2021. 6

    2021

  33. [41]

    Svd-llm: Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378,

    arXiv

  34. [42]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

  35. [43]

    Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router

    Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013, 2024. 3

    arXiv

  36. [44]

    Moe-infinity: Activation-aware expert offloading for efficient moe serving, 2024

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Activation-aware expert offloading for efficient moe serving, 2024. 3

    2024

  37. [45]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  38. [46]

    Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.01016, 2024. 2, 3, 9

    2024 arXiv

  39. [47]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019. 7

    2019

  40. [48]

    HyperMoE: Towards better mixture of experts via transferring among experts

    Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, and Jie Fu. HyperMoE: Towards better mixture of experts via transferring among experts. August 2024. 3 12

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.