Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

PuzzleMoE compresses Mixture-of-Experts models by 50% by merging expert pairs element-wise and hiding masks and signs inside bfloat16 exponent bits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:34 UTC pith:EJ7G7GSL

load-bearing objection Entry-wise dual-mask expert merging is a real, well-tested contribution; the efficiency claims and bit-packing validation need correction before the paper is release-ready. the 2 major comments →

arxiv 2511.04805 v2 pith:EJ7G7GSL submitted 2025-11-06 cs.LG cs.AI

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Yushu Zhao , Zheng Wang , Minjia Zhang This is my paper
classification cs.LG cs.AI
keywords Mixture-of-ExpertsMoE compressionexpert mergingsparse masksbit-packingbfloat16post-training compressionGPU inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PuzzleMoE tries to show that the biggest obstacle to running Mixture-of-Experts language models — storing all expert weights even though only a few are used per token — can be removed without retraining and with almost no accuracy loss. It merges experts in pairs at the individual weight-entry level, using one mask to find entries where the two experts agree in magnitude and another to keep the entries most important to each expert, then reconstructs each expert as its sign pattern times its mask times a shared merged weight. The masks and signs are not stored separately: they are packed into the exponent bits of bfloat16 weights, which the paper observes are mostly idle because expert weight exponents concentrate in a narrow range. A custom GPU kernel unpacks them on the fly during matrix-vector multiplication. If the method works as reported, a 47-billion-parameter model like Mixtral-8x7B can be halved in expert memory, compressed in minutes, and run on a single GPU with a 1.28x speedup, which would change the practical deployment cost of MoE models.

Core claim

On its own terms, the paper's central claim is that expert weights in MoE models split cleanly into shared entries (similar magnitudes across a pair of experts) and specialized entries (large magnitude and high activation saliency for one expert), so two binary masks plus a sign pattern are enough to reconstruct either expert from a single merged tensor. The reconstruction formula is Ŵ_i = (−1)^{S_i} ⊙ M_i ⊙ W_merged, computed element-wise. The authors claim this reconstruction is accurate enough that at 50% expert sparsity, average zero-shot accuracy drops only 0.2–1.4 points across four MoE families, and the bit-packing step that hides masks and signs in the bfloat16 exponent field introdu

What carries the argument

The key object is the element-wise reconstruction identity (Eq. 8), Ŵ_i = (−1)^{S_i} ⊙ M_i ⊙ W_merged. M_i is the union of a similarity mask M_sim (entry magnitudes agreeing within threshold τ_sim) and a saliency mask M_sal_i (weights whose magnitude times input-activation norm is larger for expert i); S_i is the sign bit pattern of expert i; W_merged is the averaged/saliency-selected magnitude tensor. The second half of the machinery is a bfloat16 bit-packing scheme: because expert weight exponents concentrate around 112–128, all exponents are shifted down by 112 (with values below 112 clamped), fitting into 5 bits and leaving 3 exponent bits plus the sign bit free; two mask bits and one si

Load-bearing premise

The whole no-overhead trick depends on merged expert weights using only a narrow range of exponent values — roughly values 112 to 143 in bfloat16 — so that after clamping and shifting, the freed bits really are free; if a model's weights fall outside that range, the stored masks and signs get corrupted or small weights get destroyed.

What would settle it

For any target MoE model, compute the exponent histogram of each merged expert tensor after 50% merging; if the minimum exponent is below 112 or the maximum after subtracting 112 exceeds 31 (i.e., original exponent above 143), then the bit-packing is lossy. Then run WikiText-2 perplexity for the packed vs unpacked version; any difference above 1e-3 contradicts the paper's 'no perplexity degradation' claim. The paper only reports this check for Mixtral and DeepSeek-MoE, not for the Qwen models.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At 50% expert sparsity, Mixtral-8x7B fits on one A100-80GB and Qwen3-MoE-30B fits on one A100-40GB, instead of two GPUs.
  • Compression is training-free and takes minutes (2 minutes for Mixtral, 10 for DeepSeek-MoE), so the method supports rapid re-compression for different deployments.
  • Calibration data choice barely matters: C4 and MATH give nearly identical downstream accuracy, so task-specific calibration is not needed.
  • The merge step composes with quantization: 3-bit quantization of merged weights yields about 4.8x total compression with around 1.7% max accuracy drop, so it is a complementary rather than competing method.
  • On reasoning tasks, 25% sparsity keeps Qwen3-MoE within about 12 points of baseline on AIME24 and within 1 point on Math-500, compared with baseline collapse to zero for coarse merging.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The exponent-bit packing is a general data-layout idea, not MoE-specific; the same freed-bits trick could store per-tensor metadata (masks, signs, even small calibration constants) inside any bfloat16 tensor whose dynamic range is narrow, which suggests applications to dense pruning or multi-task model merging.
  • Since random pairwise grouping is nearly as good as the searched grouping, the method's headroom likely sits in the grouping strategy rather than the merge rule; a cheap affinity-based grouping (e.g., using router statistics already collected during calibration) could recover the small observed gain without a search.
  • The fixed shift of 112 is a brittle constant across model generations; a natural testable extension is per-layer or per-tensor learned shifts selected at compression time, which would let the packing adapt to models with wider exponent ranges.
  • The method's accuracy at 50% sparsity on Mixtral (72.6 average vs 74.1 full) suggests that expert-level redundancy in MoE is much larger than previously assumed, implying that future MoE training could intentionally produce mergeable experts to make compression a first-class goal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. PuzzleMoE proposes a training-free, task-agnostic MoE compression method. Pairs of experts are merged element-wise using a similarity mask and an activation-saliency mask; the merged weights are stored in bfloat16, with the mask and sign bits packed into the freed exponent/sign bits, and a custom GEMV kernel decodes them on the fly. Experiments on four MoE models (Mixtral-8x7B, DeepSeek-MoE, Qwen1.5-MoE-A2.7B, Qwen3-MoE-30B-A3B) across WikiText-2, seven zero-shot tasks, and math benchmarks report near-baseline accuracy at 25% and 50% expert sparsity, outperforming prior merging/dropping methods by up to 16.7 MMLU points, with compression taking minutes and inference speedup of 1.19–1.28x.

Significance. If the reported results are reproducible, PuzzleMoE is a practically important contribution: it is training-free, uses a single C4 calibration pass, shows consistently small accuracy drops across four different MoE families, and avoids storing mask/sign metadata. The evaluation is stronger than typical for this area: 16 seeds with reported standard deviations, code release, and several baselines. The core merging equations are coherent. The main technical risk is the bit-packing validity on two of the four models, and the abstract contains an unsupported 45x compression-speedup number.

major comments (2)
  1. [§3.2.2, Table 1, Appendix B.1] The metadata-free 50% compression claim is load-bearing and depends on every exponent of the merged tensor W_merged (Eq. 7) lying in [112,143] after shifting, so Algorithm 1 can pack two masks and two signs into freed bits. Table 1 validates packing only for Mixtral-8x7B and DeepSeek-MoE via WikiText-2 perplexity. For Qwen1.5-MoE and Qwen3-MoE, Appendix B.1 shows only exponent plots of original expert weights, not numeric before/after packing perplexity or task accuracy, and not exponent statistics of the merged tensors. Merging can change tail statistics, so the Table 2 Qwen results do not yet establish that the bit-packed artifact is lossless. Please report before/after packing perplexity and accuracy for all four models, the count/percentage of W_merged exponents outside [112,143], and state explicitly whether Table 2 used packed or un-packed inference.
  2. [§4.4, Fig. 4(a), abstract and contribution bullet] The '45x faster compression' claim is not supported by the reported data. The text states PuzzleMoE takes 2 minutes for Mixtral-8x7B and D2 takes 55 minutes, i.e. a 27.5x reduction (55/2), not 45x. No other timing pair in Fig. 4(a) yields 45x. Please correct the claim or supply the measurement that supports 45x.
minor comments (5)
  1. [Table 4] The passage says PuzzleMoE retains 99%, 92%, and 84% of baseline accuracy on Math-500, AIME24, and AIME25. The actual ratios are 96.2/97.2=99.0%, 71.1/83.3=85.4%, and 61.5/72.9=84.4%, respectively. The 92% figure for AIME24 should be corrected to about 85%.
  2. [Algorithm 1, Fig. 3] The bit positions for the packed masks/signs ('expert pos' mapping to bits 13/12 and 15/14) are not defined in the text; the diagram is hard to read. Please add a clear bit-layout table or explanation.
  3. [§5] Typo: 'highly attribute to' should be 'highly attributed to'.
  4. [Appendix B.2] Saying that negligible correlation implies the weight entries 'can be treated as independent' is not rigorous. Since this section is an explanatory post-hoc analysis, rephrase as a modeling assumption rather than a conclusion from the correlation values.
  5. [Table 1 caption] Spelling of 'DeepSeek' is inconsistent ('Deepseek-MoE' appears in several places). Please unify.

Circularity Check

0 steps flagged

No significant circularity: merging is an explicit construction, accuracy is evaluated on held-out tasks, and the unvalidated bit-packing on Qwen is a verification gap, not a circular reduction.

full rationale

PuzzleMoE's derivation chain is self-contained and does not reduce any prediction to its inputs by construction. Equations (1)-(8) define a merging procedure: W_merged is explicitly constructed from |W_i| and |W_j| via similarity and saliency masks, and the decode rule (8) recovers each expert's masked weight from that same construction, a design definition rather than a fitted prediction. Downstream accuracy is measured on held-out benchmarks (ARC, HellaSwag, PIQA, BoolQ, Wino, MMLU, GSM8K) after fixing the hyperparameter tau_sim = 0.4 in Section 5; the Wikitext-2 sweep used to select tau_sim is validation/tuning, and the paper's central claims rest on seven held-out tasks plus math benchmarks, not on the tuning metric. Appendix B.2 is explicitly a post-hoc probability analysis meant to 'support this design', not to fit any constant. The bit-packing scheme is verified numerically on Mixtral-8x7B and DeepSeek-MoE in Table 1; the absence of a numeric before/after packing perplexity for Qwen1.5-MoE and Qwen3-MoE (only exponent plots in Appendix B.1) is a missing-evidence or correctness risk for the 'no metadata overhead' claim on those models, but it is not circular because no prediction reduces to a fit. Self-citations are background only and not load-bearing. No uniqueness theorem is invoked. Therefore the accuracy, compression, and speedup claims are independently grounded.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central method relies on the empirical validity of entry-wise similarity/saliency as a proxy for shared vs. specialized knowledge, and on the narrow exponent distribution of MoE weights. The main free parameters are τ_sim and the exponent shift constant; both are chosen from validation/observational data. No new physical or model-level entities are introduced.

free parameters (2)
  • similarity threshold τ_sim = 0.4
    Tuned via WikiText-2 perplexity sweep (Fig. 5; best 0.3–0.5 for Mixtral and DeepSeek); fixed to 0.4 for all models and tasks in the main results. This is a free hyperparameter chosen on validation data.
  • exponent shift constant (112) = 112 (clamp below to 112)
    Chosen from the observed Mixtral-8x7B exponent distribution (Fig. 3); applied to all models and layers. The constant is ad hoc and its no-loss property is only validated for Mixtral and DeepSeek (Table 1).
axioms (5)
  • domain assumption Expert weights have element-wise magnitude similarity that reflects shared knowledge, and per-entry saliency (|W|⊙||X||₂) identifies which expert's entry to preserve.
    Invoked throughout §3.1 (Eqs. 2, 4–7). This is the core ML premise of the merging algorithm; it is validated only empirically by downstream accuracy, not by an independent mechanism.
  • domain assumption The bfloat16 exponent field of merged expert weights can be shifted by a fixed constant (112) and clamped below without loss, so 4 metadata bits fit in the freed bits.
    Invoked in §3.2.2 and Algorithm 1. Validated only for Mixtral-8x7B and DeepSeek-MoE on WikiText-2 (Table 1); Qwen models lack before/after packing perplexity.
  • domain assumption Wanda-style activation saliency computed on 128 C4 samples (seq len 2048) is sufficient to determine per-entry preservation for all downstream tasks.
    Eq. (4); robustness to calibration data shown in Table 5 (C4 vs MATH) but only for Mixtral. Other models are not tested for calibration robustness.
  • standard math Expert weight entries can be modeled as independent zero-mean Gaussians for the probability calculation of the similarity criterion.
    Appendix B.2 derives the ratio-of-half-normals probability under independence; this matches Qwen/Deepseek correlations (~0) but deviates for Mixtral (ρ=0.26). This is explanatory and not load-bearing for the method's correctness.
  • domain assumption Random pairwise grouping of experts is sufficient; search-based grouping yields negligible gains.
    Table 7 shows searched grouping improves Mixtral 72.6→72.9 avg acc and Deepseek 62.1→62.4; random grouping is used by default. If grouping were adversarial, results could degrade, but the ablation bounds this risk.

pith-pipeline@v1.3.0-alltime-deepseek · 21086 in / 22536 out tokens · 157856 ms · 2026-08-03T23:34:12.487409+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies, yet they often suffer from performance drop at high compression ratios. In this paper, we introduce PuzzleMoE, a training-free MoE compression method that achieves both high accuracy and efficient inference through two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It uses a dual-mask to capture both shared and expert-specific parameters. Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE can compress MoE models by up to 50% while maintaining accuracy across various tasks. Specifically, it outperforms prior MoE compression methods by up to 16.7% on MMLU at 50% compression ratio, and achieves up to 1.28\times inference speedup.

Figures

Figures reproduced from arXiv: 2511.04805 by Minjia Zhang, Yushu Zhao, Zheng Wang.

Figure 1
Figure 1. Figure 1: (a): Accuracy of different MoE models on MMLU benchmark under 50%compression ra [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Procedure of sparse expert merging algorithm. We merge two experts by computing [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the mask packing procedure. (a): the distribution of Bfloat16 weight ex [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: System performance for different MoE models and tasks. (a): Compression time compar [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a): Wikitext2 perplexity of Mixtral and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Exponent distribution of different MoE models. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Relation between similarity ratio and similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Combining PuzzleMoE with quantization. B.4 DETAILED RESULTS OF PUZZLEMOE WITH DIFFERENT SEEDS The detailed accuracy results of PuzzleMoE with different random seeds on Mixtral-8x7B, Deepseek-MoE, Qwen1.5-MoE, and Qwen3-MoE are shown in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Pruning and Distilling Mixture-of-Experts into Dense Language Models

    cs.CL 2026-05 unverdicted novelty 6.0

    A systematic MoE-to-dense conversion via expert scoring, grouping, and distillation yields +6.3 pp average accuracy over dense-to-dense pruning at matched parameter count on tested models.

  2. ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits

    cs.LG 2026-01 reject novelty 6.0

    Experts in an MoE are reparameterized as butterfly rotations of a single shared ternary matrix, giving O(d^2 + N d log d) memory and a claimed ~150x compression at 256 experts.

Reference graph

Works this paper leans on

39 extracted references · 28 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641

    Pith/arXiv arXiv 2019

  2. [2]

    Retraining-free merging of sparse moe via hierarchical clustering, 2025

    I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee. Retraining-free merging of sparse moe via hierarchical clustering, 2025. URL https://arxiv.org/abs/2410.08589

    arXiv 2025

  3. [3]

    Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044

    Pith/arXiv arXiv 2019

  4. [4]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    Pith/arXiv arXiv 2018

  5. [5]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. URL https://arxiv.org/abs/2401.06066

    Pith/arXiv arXiv 2024

  6. [6]

    Mxmoe: Mixed-precision quantization for moe with accuracy and performance co-design, 2025

    Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Mxmoe: Mixed-precision quantization for moe with accuracy and performance co-design, 2025. URL https://arxiv.org/abs/2505.05799

    Pith/arXiv arXiv 2025

  7. [7]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/2210.17323

    Pith/arXiv arXiv 2023

  8. [8]

    Delta decompression for moe-based llms compression, 2025

    Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based llms compression, 2025. URL https://arxiv.org/abs/2502.17298

    Pith/arXiv arXiv 2025

  9. [9]

    Towards efficient mixture of experts: A holistic study of compression techniques, 2025 a

    Shwai He, Daize Dong, Liang Ding, and Ang Li. Towards efficient mixture of experts: A holistic study of compression techniques, 2025 a . URL https://arxiv.org/abs/2406.02500

    Pith/arXiv arXiv 2025

  10. [10]

    Localize-and-stitch: Efficient model merging via sparse task arithmetic, 2025 b

    Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stitch: Efficient model merging via sparse task arithmetic, 2025 b . URL https://arxiv.org/abs/2408.13656

    Pith/arXiv arXiv 2025

  11. [11]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  12. [12]

    Moequant: Enhancing quantization for mixture-of-experts large language models via expert-balanced sampling and affinity guidance, 2025

    Xing Hu, Zhixuan Chen, Dawei Yang, Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. Moequant: Enhancing quantization for mixture-of-experts large language models via expert-balanced sampling and affinity guidance, 2025. URL https://arxiv.org/abs/2505.03804

    Pith/arXiv arXiv 2025

  13. [13]

    Milo: Efficient quantized moe inference with mixture of low-rank compensators, 2025

    Beichen Huang, Yueming Yuan, Zelei Shao, and Minjia Zhang. Milo: Efficient quantized moe inference with mixture of low-rank compensators, 2025. URL https://arxiv.org/abs/2504.02658

    Pith/arXiv arXiv 2025

  14. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven...

    Pith/arXiv arXiv 2024

  15. [15]

    Mahoney, and Kurt Keutzer

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2024. URL https://arxiv.org/abs/2306.07629

    Pith/arXiv arXiv 2024

  16. [16]

    Compressed sparse tiles for memory-efficient unstructured and semi-structured sparsity

    Mike Lasby, Max Zimmer, Sebastian Pokutta, and Erik Schultheis. Compressed sparse tiles for memory-efficient unstructured and semi-structured sparsity. In Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference, 2025. URL https://openreview.net/forum?id=iso0KV2HVq

    2025

  17. [17]

    Haeun Lee, Omin Kwon, Yeonhong Park, and Jae W. Lee. Nestedfp: High-performance, memory-efficient dual-precision floating point support for llms, 2025 a . URL https://arxiv.org/abs/2506.02024

    arXiv 2025

  18. [18]

    Stun: Structured-then-unstructured pruning for scalable moe pruning, 2025 b

    Jaeseong Lee, seung-won hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, and Yuxiong He. Stun: Structured-then-unstructured pruning for scalable moe pruning, 2025 b . URL https://arxiv.org/abs/2409.06211

    Pith/arXiv arXiv 2025

  19. [19]

    Sub-moe: Efficient mixture-of-expert llms compression via subspace expert merging, 2025

    Lujun Li, Zhu Qiyuan, Jiacheng Wang, Wei Li, Hao Gu, Sirui Han, and Yike Guo. Sub-moe: Efficient mixture-of-expert llms compression via subspace expert merging, 2025. URL https://arxiv.org/abs/2506.23266

    Pith/arXiv arXiv 2025

  20. [20]

    Merge, then compress: Demystify efficient smoe with hints from its routing policy, 2024

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy, 2024. URL https://arxiv.org/abs/2310.01334

    Pith/arXiv arXiv 2024

  21. [21]

    Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. URL https://arxiv.org/abs/2306.00978

    Pith/arXiv arXiv 2024

  22. [22]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models, 2024

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models, 2024. URL https://arxiv.org/abs/2402.14800

    Pith/arXiv arXiv 2024

  23. [23]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  24. [24]

    Ronald Miller

    H. Ronald Miller. Optimization: Foundations and Applications. John Wiley & Sons, New York, 2011. ISBN 978-1-118-03118-6

    2011

  25. [25]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    Pith/arXiv arXiv 2023

  26. [26]

    Winogrande: An adversarial winograd schema challenge at scale, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.10641

    Pith/arXiv arXiv 2019

  27. [27]

    Everything you always wanted to know about storage compressibility of pre-trained ml models but were afraid to ask, 2024

    Zhaoyuan Su, Ammar Ahmed, Zirui Wang, Ali Anwar, and Yue Cheng. Everything you always wanted to know about storage compressibility of pre-trained ml models but were afraid to ask, 2024. URL https://arxiv.org/abs/2402.13429

    Pith/arXiv arXiv 2024

  28. [28]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024. URL https://arxiv.org/abs/2306.11695

    Pith/arXiv arXiv 2024

  29. [29]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024

    Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. URL https://qwenlm.github.io/blog/qwen-moe/

    2024

  30. [30]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  31. [31]

    Moe-i ^2 : Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition, 2024 a

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i ^2 : Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition, 2024 a . URL https://arxiv.org/abs/2411.01016

    Pith/arXiv arXiv 2024

  32. [32]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities, 2024 b

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities, 2024 b . URL https://arxiv.org/abs/2408.07666

    Pith/arXiv arXiv 2024

  33. [33]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    Pith/arXiv arXiv 2019

  34. [34]

    70 URL https://arxiv.org/abs/2504.11651

    Tianyi Zhang, Yang Sui, Shaochen Zhong, Vipin Chaudhary, Xia Hu, and Anshumali Shrivastava. 70 URL https://arxiv.org/abs/2504.11651

    arXiv

  35. [35]

    Low-rank and sparse model merging for multi-lingual speech recognition and translation, 2025

    Qiuming Zhao, Guangzhi Sun, and Chao Zhang. Low-rank and sparse model merging for multi-lingual speech recognition and translation, 2025. URL https://arxiv.org/abs/2502.17380

    Pith/arXiv arXiv 2025

  36. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  37. [37]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  38. [38]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  39. [39]

    However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2048