Pith. sign in

REVIEW 10 cited by

Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.02500 v3 pith:CKCR5UUC submitted 2024-06-04 cs.LG cs.AI

Shwai He , Daize Dong , Liang Ding , Ang Li This is my paper
classification cs.LGcs.AI
keywords expertexpertsperformancetrimmingdropmixturetechniqueswhile
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deployment. The Mixture of Experts (MoE) architecture offers a promising solution by dynamically selecting and activating only a subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE introduces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, we propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model performance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts to further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods-Layer Drop and Block Drop-along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving a 6.05x speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8x7B. Our code is released at https://github.com/CASE-Lab-UMD/Unified-MoE-Compression.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    HodgeCover isolates the harmonic kernel of a simplicial Laplacian on an expert 2-complex to identify irreducible merge cycles and selects experts for aggressive compression, matching or exceeding baselines on open-wei...

  2. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    cs.LG 2026-03 conditional novelty 7.0 of 10

    EvoESAP uses evolutionary search guided by a speculative-decoding-inspired ESAP metric to discover non-uniform layer-wise sparsity allocations for MoE expert pruning, improving generation accuracy up to 19.6% at 50% sparsity.

  3. PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

    cs.LG 2025-11 conditional novelty 7.0 of 10

    A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.

  4. REAM: Merging Improves Pruning of Experts in LLMs

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    REAM merges experts in MoE LLMs rather than pruning them, often matching uncompressed performance by tuning the mix of calibration data.

  5. ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits

    cs.LG 2026-01 reject novelty 6.0 of 10

    Experts in an MoE are reparameterized as butterfly rotations of a single shared ternary matrix, giving O(d^2 + N d log d) memory and a claimed ~150x compression at 256 experts.

  6. Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs

    cs.CL 2025-09 conditional novelty 6.0 of 10

    DERN prunes SMoE LLMs by decomposing removed experts into neuron segments, reassigning the best-matching ones to kept experts, and clustering them into compact replacements, beating prior pruning baselines without retraining.

  7. Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning

    cs.LG 2024-11 unverdicted novelty 6.0 of 10

    CD-MoE condenses fine-grained MoE layers with shared experts into dense layers, retaining 90% accuracy with 27.5% memory cut and 1.26x speedup on DeepSeekMoE-16B, recovering 98% via brief fine-tuning.

  8. Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection

    cs.LG 2024-11 unverdicted novelty 6.0 of 10

    Lynx exploits training-induced batch-level expert activation skews via AffinityBinning to reduce invoked experts per batch, delivering up to 1.30x throughput with under 1% accuracy loss across four model families.

  9. Does a Global Perspective Help Prune Sparse MoEs Elegantly?

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    GRAPE is a global redundancy-aware pruning strategy for sparse MoEs that dynamically allocates pruning budgets across layers and improves average accuracy by 1.40% over the best local baseline across tested models and...

  10. Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning

    cs.LG 2025-09 reject novelty 3.0 of 10

    A hyperparameter search for LLM math reasoning that reports simulated, not measured, performance gains.

Pith tools